Skip to content

โœจ feat: add search1api crawler implementation for WeChat Sogou links #7036

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weโ€™ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Mar 19, 2025
Merged

โœจ feat: add search1api crawler implementation for WeChat Sogou links #7036

merged 5 commits into from
Mar 19, 2025

Conversation

fatwang2
Copy link
Contributor

๐Ÿ’ป ๅ˜ๆ›ด็ฑปๅž‹ | Change Type

  • โœจ feat
  • ๐Ÿ› fix
  • โ™ป๏ธ refactor
  • ๐Ÿ’„ style
  • ๐Ÿ‘ท build
  • โšก๏ธ perf
  • ๐Ÿ“ docs
  • ๐Ÿ”จ chore

๐Ÿ”€ ๅ˜ๆ›ด่ฏดๆ˜Ž | Description of Change

Add search1api crawler for WeChat Sogou links

๐Ÿ“ ่กฅๅ……ไฟกๆฏ | Additional Information

@vercel Vercel
Copy link

vercel bot commented Mar 19, 2025

@fatwang2 is attempting to deploy a commit to the LobeChat Desktop Team on Vercel.

A member of the Team first needs to authorize it.

@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Mar 19, 2025
@lobehubbot
Copy link
Member

๐Ÿ‘ @fatwang2

Thank you for raising your pull request and contributing to our Community
Please make sure you have followed our contributing guidelines. We will review it as soon as possible.
If you encounter any problems, please feel free to connect with us.
้žๅธธๆ„Ÿ่ฐขๆ‚จๆๅ‡บๆ‹‰ๅ–่ฏทๆฑ‚ๅนถไธบๆˆ‘ไปฌ็š„็คพๅŒบๅšๅ‡บ่ดก็Œฎ๏ผŒ่ฏท็กฎไฟๆ‚จๅทฒ็ป้ตๅพชไบ†ๆˆ‘ไปฌ็š„่ดก็ŒฎๆŒ‡ๅ—๏ผŒๆˆ‘ไปฌไผšๅฐฝๅฟซๅฎกๆŸฅๅฎƒใ€‚
ๅฆ‚ๆžœๆ‚จ้‡ๅˆฐไปปไฝ•้—ฎ้ข˜๏ผŒ่ฏท้šๆ—ถไธŽๆˆ‘ไปฌ่”็ณปใ€‚

@dosubot dosubot bot added the ๐ŸŒ  Feature Request New feature or request | ็‰นๆ€งไธŽๅปบ่ฎฎ label Mar 19, 2025
@gru-agent Gru Agent
Copy link
Contributor

gru-agent bot commented Mar 19, 2025 •
edited
Loading

TestGru Assignment

Summary

Link CommitId Status Reason
Detail 7ce9690 โœ… Finished

Files

File Pull Request
packages/web-crawler/src/crawImpl/search1api.ts โŒ Failure (I failed to write the unit tests for the file.)

Tip

You can @gru-agent and leave your feedback. TestGru will make adjustments based on your input

arvinxx
arvinxx reviewed Mar 19, 2025
View reviewed changes
Copy link
Member

@arvinxx arvinxx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ๅ…ถไป–ๆฒก้—ฎ้ข˜~

@codecov Codecov
Copy link

codecov bot commented Mar 19, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 89.07563% with 13 lines in your changes missing coverage. Please review.

Project coverage is 91.40%. Comparing base (cd0828d) to head (99f4d87).
Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
packages/web-crawler/src/utils/withTimeout.ts 50.00% 10 Missing โš ๏ธ
packages/web-crawler/src/crawImpl/search1api.ts 97.59% 2 Missing โš ๏ธ
packages/web-crawler/src/crawImpl/naive.ts 50.00% 1 Missing โš ๏ธ
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #7036      +/-   ##
==========================================
+ Coverage   91.38%   91.40%   +0.02%     
==========================================
  Files         736      738       +2     
  Lines       69677    69779     +102     
  Branches     4787     4804      +17     
==========================================
+ Hits        63672    63781     +109     
+ Misses       6005     5998       -7
Flag Coverage ฮ”
app 91.40% <89.07%> (+0.02%) โฌ†๏ธ
server 97.54% <รธ> (รธ)

Flags with carried forward coverage won't be shown. Click here to find out more.

โ˜” View full report in Codecov by Sentry.
๐Ÿ“ข Have feedback on the report? Share it here.

๐Ÿš€ New features to boost your workflow:
  • โ„ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • ๐Ÿ“ฆ JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@fatwang2 fatwang2 changed the title feat: add search1api crawler implementation for WeChat Sogou links โœจ feat: add search1api crawler implementation for WeChat Sogou links Mar 19, 2025
@hezhijie0327
Copy link
Contributor

hezhijie0327 commented Mar 19, 2025 •
edited
Loading

ๅ“‡ ๅŽŸๆฅ search1api ๅฏไปฅ่งฃๅ†ณ้ชŒ่ฏ็ ็š„้—ฎ้ข˜

่ฆไนˆๆŠŠ https://sogou.com/link?url= ไนŸๅŠ ไธ‹๏ผŸ
ไน‹ๅ‰็ป™ SearXNG ้€‚้… Sogou ็š„ๆ—ถๅ€™๏ผŒๅฐฑๅ‘็Žฐ Sogou ็ณปๅˆ—้ƒฝๆ˜ฏๅŠ ๅฏ†้“พๆŽฅ...

image

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. ๐Ÿ‘ฏ๐Ÿ‘ญ๐Ÿป๐Ÿง‘โ€๐Ÿคโ€๐Ÿง‘๐Ÿ‘ซ๐Ÿง‘๐Ÿฟโ€๐Ÿคโ€๐Ÿง‘๐Ÿป๐Ÿ‘ฉ๐Ÿพโ€๐Ÿคโ€๐Ÿ‘จ๐Ÿฟ๐Ÿ‘ฌ๐Ÿฟ

Wow, it turns out that search1api can solve the problem of verification code

Or add https://sogou.com/link?url=?
When I adapted Sogou to SearXNG, I found that the Sogou series is full of encrypted links...

@fatwang2
Copy link
Contributor Author

็›ฎๅ‰้€‚้…ไบ†ๆœ็‹—ๅพฎไฟก๏ผŒ่ฟ˜ๆฒก้€‚้…ๆœ็‹—๏ผŒๆˆ‘ๆต‹่ฏ•ไธ‹

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. ๐Ÿ‘ฏ๐Ÿ‘ญ๐Ÿป๐Ÿง‘โ€๐Ÿคโ€๐Ÿง‘๐Ÿ‘ซ๐Ÿง‘๐Ÿฟโ€๐Ÿคโ€๐Ÿง‘๐Ÿป๐Ÿ‘ฉ๐Ÿพโ€๐Ÿคโ€๐Ÿ‘จ๐Ÿฟ๐Ÿ‘ฌ๐Ÿฟ

Currently, I have adapted to Sogou WeChat, but I haven't adapted to Sogou yet. I'll test it

@fatwang2
Copy link
Contributor Author

ไนŸ้€‚้…ไบ†ๆœ็‹—็š„้“พๆŽฅ

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. ๐Ÿ‘ฏ๐Ÿ‘ญ๐Ÿป๐Ÿง‘โ€๐Ÿคโ€๐Ÿง‘๐Ÿ‘ซ๐Ÿง‘๐Ÿฟโ€๐Ÿคโ€๐Ÿง‘๐Ÿป๐Ÿ‘ฉ๐Ÿพโ€๐Ÿคโ€๐Ÿ‘จ๐Ÿฟ๐Ÿ‘ฌ๐Ÿฟ

Also adapted to Sogou link

@fatwang2
Copy link
Contributor Author

ๆˆ‘่กฅๅ……่ฏดๆ˜Žไธ‹๏ผŒๆœ็‹—็š„ๅ็ˆฌๆฏ”ๆœ็‹—ๅพฎไฟก่ฆ็ฎ€ๅ•ๅพˆๅคš๏ผŒๆˆ‘ๅˆšๆ‰่ฏ•ไบ†ๅ‡ ไธช็ˆฌๅ–ๆœๅŠก๏ผŒๆœ็‹—ๆœ‰็š„ๆ˜ฏๅฏไปฅ็ˆฌๅ–็š„๏ผŒไฝ†ๆ˜ฏๆœ็‹—ๅพฎไฟกๆ˜ฏๅ…จ้ƒจไธ่กŒ

@lobehubbot
Copy link
Member

Bot detected the issue body's language is not English, translate it automatically. ๐Ÿ‘ฏ๐Ÿ‘ญ๐Ÿป๐Ÿง‘โ€๐Ÿคโ€๐Ÿง‘๐Ÿ‘ซ๐Ÿง‘๐Ÿฟโ€๐Ÿคโ€๐Ÿง‘๐Ÿป๐Ÿ‘ฉ๐Ÿพโ€๐Ÿคโ€๐Ÿ‘จ๐Ÿฟ๐Ÿ‘ฌ๐Ÿฟ

Let me add that Sogou's anti-crawling is much simpler than Sogou WeChat. I just tried a few crawling services. Some of Sogou can crawl, but Sogou WeChat doesn't work.

@arvinxx arvinxx merged commit 7327138 into lobehub:main Mar 19, 2025
9 of 12 checks passed
@lobehubbot
Copy link
Member

โค๏ธ Great PR @fatwang2 โค๏ธ

The growth of project is inseparable from user feedback and contribution, thanks for your contribution! If you are interesting with the lobehub developer community, please join our discord and then dm @arvinxx or @canisminor1990. They will invite you to our private developer channel. We are talking about the lobe-chat development or sharing ai newsletter around the world.
้กน็›ฎ็š„ๆˆ้•ฟ็ฆปไธๅผ€็”จๆˆทๅ้ฆˆๅ’Œ่ดก็Œฎ๏ผŒๆ„Ÿ่ฐขๆ‚จ็š„่ดก็Œฎ! ๅฆ‚ๆžœๆ‚จๅฏน LobeHub ๅผ€ๅ‘่€…็คพๅŒบๆ„Ÿๅ…ด่ถฃ๏ผŒ่ฏทๅŠ ๅ…ฅๆˆ‘ไปฌ็š„ discord๏ผŒ็„ถๅŽ็งไฟก @arvinxx ๆˆ– @canisminor1990ใ€‚ไป–ไปฌไผš้‚€่ฏทๆ‚จๅŠ ๅ…ฅๆˆ‘ไปฌ็š„็งๅฏ†ๅผ€ๅ‘่€…้ข‘้“ใ€‚ๆˆ‘ไปฌๅฐ†ไผš่ฎจ่ฎบๅ…ณไบŽ Lobe Chat ็š„ๅผ€ๅ‘๏ผŒๅˆ†ไบซๅ’Œ่ฎจ่ฎบๅ…จ็ƒ่Œƒๅ›ดๅ†…็š„ AI ๆถˆๆฏใ€‚

github-actions bot pushed a commit that referenced this pull request Mar 19, 2025
@semantic-release-bot
๐Ÿ”– chore(release): v1.73.0 [skip ci]
ce03db6 
## [Version&nbsp;1.73.0](v1.72.1...v1.73.0)
<sup>Released on **2025-03-19**</sup>

#### โœจ Features

- **misc**: Add Cohere provider support, add search1api crawler implementation for WeChat Sogou links.

<br/>

<details>
<summary><kbd>Improvements and Fixes</kbd></summary>

#### What's improved

* **misc**: Add Cohere provider support, closes [#7016](#7016) ([2a4e2ed](2a4e2ed))
* **misc**: Add search1api crawler implementation for WeChat Sogou links, closes [#7036](#7036) ([7327138](7327138))

</details>

<div align="right">

[![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top)

</div>
@lobehubbot
Copy link
Member

๐ŸŽ‰ This PR is included in version 1.73.0 ๐ŸŽ‰

The release is available on:

Your semantic-release bot ๐Ÿ“ฆ๐Ÿš€

github-actions bot pushed a commit to jaworldwideorg/OneJA-Bot that referenced this pull request Mar 20, 2025
@semantic-release-bot
๐Ÿ”– chore(release): v1.73.0 [skip ci]
336957e 
## [Version&nbsp;1.73.0](v1.72.3...v1.73.0)
<sup>Released on **2025-03-20**</sup>

#### โœจ Features

- **misc**: Add Cohere provider support, add search1api crawler implementation for WeChat Sogou links.

<br/>

<details>
<summary><kbd>Improvements and Fixes</kbd></summary>

#### What's improved

* **misc**: Add Cohere provider support, closes [lobehub#7016](https://github.com/jaworldwideorg/OneJA-Bot/issues/7016) ([2a4e2ed](2a4e2ed))
* **misc**: Add search1api crawler implementation for WeChat Sogou links, closes [lobehub#7036](https://github.com/jaworldwideorg/OneJA-Bot/issues/7036) ([7327138](7327138))

</details>

<div align="right">

[![](https://img.shields.io/badge/-BACK_TO_TOP-151515?style=flat-square)](#readme-top)

</div>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@arvinxx arvinxx arvinxx left review comments

+1 more reviewer

@hezhijie0327 hezhijie0327 hezhijie0327 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

๐ŸŒ  Feature Request New feature or request | ็‰นๆ€งไธŽๅปบ่ฎฎ released size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@fatwang2 @lobehubbot @hezhijie0327 @arvinxx