Skip to content

Commit 0dbbf7d

Browse files
committed
🔎 feat: Native Web Search with Citation References (#7516)
* WIP: search tool integration * WIP: Add web search capabilities and API key management to agent actions * WIP: web search capability to agent configuration and selection * WIP: Add web search capability to backend agent configuration * WIP: add web search option to default agent form values * WIP: add attachments for web search * feat: add plugin for processing web search citations * WIP: first pass, Citation UI * chore: remove console.log * feat: Add AnimatedTabs component for tabbed UI functionality * refactor: AnimatedTabs component with CSS animations and stable ID generation * WIP example content * feat: SearchContext for managing search results apart from MessageContext * feat: Enhance AnimatedTabs with underline animation and state management * WIP: first pass, Implement dynamic tab functionality in Sources component with search results integration * fix: Update class names for improved styling in Sources and AnimatedTabs components * feat: Improve styling and layout in Sources component with enhanced button and item designs * feat: Refactor Sources component to integrate OGDialog for source display and improve layout * style: Update background color in SourceItem and SourcesGroup components for improved visibility * refactor: Sources component to enhance SourceItem structure and improve favicon handling * style: Adjust font size of domain text in SourceItem for better readability * feat: Add localization for citation source and details in CompositeCitation component * style: add theming to Citation components * feat: Enhance SourceItem component with dialog support and improved hovercard functionality * feat: Add localization for sources tab and image alt text in Sources component * style: Replace divs with spans for better semantic structure in CompositeCitation and Citation components * refactor: Sources component to use useMemo for tab generation and improve performance * chore: bump @librechat/agents to v2.4.318 * chore: update search result types * fix: search results retrieval in ContentParts component, re-render attachments when expected * feat: update sources style/types to use latest search result structure * style: enhance Dialog (expanded) SourceItem component with link wrapping and improved styling * style: update ImageItem component styling for improved title visibility * refactor: remove SourceItemBase component and adjust SourceItem layout for improved styling * chore: linting twcss order * fix: prevent FileAttachment from rendering search attachments * fix: append underscore to responseMessageId for unique identification to prevent mapping of previous latest message's attachments * chore: remove unused parameter 'useSpecs' from loadTools function * chore: twcss order * WIP: WebSearch Tool UI * refactor: add limit parameter to StackedFavicons for customizable source display * refactor: optimize search results memoization by making more granular and separate conerns * refactor: integrated StackedFavicons to WebSearch mid-run * chore: bump @librechat/agents to expose handleToolCallChunks * chore: use typedefs from dedicated file instead of defining them in AgentClient module * WIP: first pass, search progress results * refactor: move createOnSearchResults function to a dedicated search module * chore: bump @librechat/agents to v2.4.320 * WIP: first pass, search results processed UX * refactor: consolidate context variables in createOnSearchResults function * chore: bump @librechat/agents to v2.4.321 * feat: add guidelines for web search tool response formatting in loadTools function * feat: add isLast prop to Part component and update WebSearch logic for improved state handling * style: update Hovercard styles for improved UI consistency * feat: export FaviconImage component for improved accessibility in other modules * refactor: export getCleanDomain function and use FaviconImage in Citation component for improved source representation * refactor: implement SourceHovercard component for consistency and DRY compliance * fix: replace <p> with <span> for snippet and title in SourceItem and SourceHovercard for consistency * style: `not-prose` * style: remove 'not-prose' class for consistency in SourceItem, Citation, and SourceHovercard components, adjust style classes * refactor: `imageUrl` on hover and prevent duplicate sources * refactor: enhance SourcesGroup dialog layout and improve source item presentation * refactor: reorganize Web Components, save in same directory * feat: add 'news' refType to refTypeMap for citation sources * style: adjust Hovercard width for improved layout * refactor: update tool usage guidelines for improved clarity and execution * chore: linting * feat: add Web Search badge with initial permissions and local storage logic * feat: add webSearch support to interface and permissions schemas * feat: implement Web Search API key management and localization updates * feat: refactor Web Search API key handling and integrate new search API key form * fix: remove unnecessary visibility state from FileAttachment component * feat: update WebSearch component to use Globe icon and localized search label * feat: enhance ApiKeyDialog with dropdown for reranker selection and update translations * feat: implement dropdown menus for engine, scraper, and reranker selection in ApiKeyDialog * chore: linting and add unknown instead of `any` type * feat: refactor ApiKeyDialog and useAuthSearchTool for improved API key management * refactor: update ocrSchema to use template literals for default apiKey and baseURL * feat: add web search configuration and utility functions for environment variable extraction * fix: ensure filepath is defined before checking its prefix in useAttachmentHandler * feat: enhance web search functionality with improved configuration and environment variable extraction for authFields * fix: update auth type in TPluginAction and TUpdateUserPlugins to use Partial<Record<string, string>> * feat: implement web search authentication verification and enhance webSearchAuth structure * feat: enhance ephemeral agent handling with new web search capability and type definition * feat: enhance isEphemeralAgent function to include web search selection * feat: refactor verifyWebSearchAuth to improve key handling and authentication checks * feat: implement loadWebSearchAuth function for improved web search authentication handling * feat: enhance web search authentication with new configuration options and refactor related types * refactor: rename search engine to search provider and update related localization keys * feat: update verifyWebSearchAuth to handle multiple authentication types and improve error handling * feat: update ApiKeyDialog to accept authTypes prop and remove isUserProvided check * feat: add tests for extractWebSearchEnvVars and loadWebSearchAuth functions * feat: enhance loadWebSearchAuth to support specific service checks for providers, scrapers, and rerankers * fix: update web search configuration key and adjust auth result handling in loadTools function * feat: add new progress key for repeated web searching and update localization * chore: bump @librechat/agents to 2.4.322 * feat: enhance loadTools function to include ISO time and improve search tool logging * feat: update StackedFavicons to handle negative start index and improve citation attribution styling and text * chore: update .gitignore to categorize AI-related files * fix: mobile responsiveness of sources/citations hovercards * feat: enhance source display with improved line clamping for better readability * chore: bump @librechat/agents to v2.4.33 * feat: add handling for image sources in references mapping * chore: bump librechat-data-provider version to 0.7.84 * chore: bump @librechat/agents version to 2.4.34 * fix: update auth handling to support multiple auth types in tools and allow key configuration in agent panel * chore: remove redundant agent attribution text from search form * fix: web search auth uninstall * refactor: convert CheckboxButton to a forwardRef component and update setValue callback signature * feat: add triggerRef prop to ApiKeyDialog components for improved dialog control * feat: integrate triggerRef in CodeInterpreter and WebSearch components for enhanced dialog management * feat: enhance ApiKeyDialog with additional links for Firecrawl and Jina API key guidance * feat: implement web search configuration handling in ApiKeyDialog and add tests for dropdown visibility * fix: update webSearchConfig reference in config route for correct payload assignment * feat: update ApiKeyDialog to conditionally render sections based on authTypes and modify loadWebSearchAuth to correctly categorize authentication types * feat: refactor ApiKeyDialog and related tests to use SearchCategories and RerankerTypes enums and remove nested ternaries * refactor: move ThinkingButton rendering to improve layout consistency in ContentParts * feat: integrate search context into Markdown component to conditionally include unicodeCitation plugin * chore: bump @librechat/agents to v2.4.35 * chore: remove unused 18n key * ci: add WEB_SEARCH permission testing and update AppService tests for new webSearch configuration * ci: add more comprehensive tests for loadWebSearchAuth to validate authentication handling and authTypes structure * chore: remove debugging console log from web.spec.ts to clean up test output
1 parent bf80cf3 commit 0dbbf7d

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

73 files changed

+15715
-11352
lines changed

api/app/clients/tools/util/handleTools.js

Lines changed: 39 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,13 @@
11
const { SerpAPI } = require('@langchain/community/tools/serpapi');
22
const { Calculator } = require('@langchain/community/tools/calculator');
3-
const { createCodeExecutionTool, EnvVar } = require('@librechat/agents');
4-
const { Tools, Constants, EToolResources } = require('librechat-data-provider');
3+
const { EnvVar, createCodeExecutionTool, createSearchTool } = require('@librechat/agents');
4+
const {
5+
Tools,
6+
Constants,
7+
EToolResources,
8+
loadWebSearchAuth,
9+
replaceSpecialVars,
10+
} = require('librechat-data-provider');
511
const { getUserPluginAuthValue } = require('~/server/services/PluginService');
612
const {
713
availableTools,
@@ -138,7 +144,6 @@ const loadTools = async ({
138144
agent,
139145
model,
140146
endpoint,
141-
useSpecs,
142147
tools = [],
143148
options = {},
144149
functions = true,
@@ -263,6 +268,37 @@ const loadTools = async ({
263268
return createFileSearchTool({ req: options.req, files, entity_id: agent?.id });
264269
};
265270
continue;
271+
} else if (tool === Tools.web_search) {
272+
const webSearchConfig = options?.req?.app?.locals?.webSearch;
273+
const result = await loadWebSearchAuth({
274+
userId: user,
275+
loadAuthValues,
276+
webSearchConfig,
277+
});
278+
const { onSearchResults, onGetHighlights } = options?.[Tools.web_search] ?? {};
279+
requestedTools[tool] = async () => {
280+
// const { files, toolContext } = await primeSearchFiles(options);
281+
// if (toolContext) {
282+
// toolContextMap[tool] = toolContext;
283+
// }
284+
toolContextMap[tool] = `# \`${tool}\`:
285+
Current Date & Time: ${replaceSpecialVars({ text: '{{iso_datetime}}' })}
286+
1. **Execute immediately without preface** when using \`${tool}\`.
287+
2. **After the search, begin with a brief summary** that directly addresses the query without headers or explaining your process.
288+
3. **Structure your response clearly** using Markdown formatting (Level 2 headers for sections, lists for multiple points, tables for comparisons).
289+
4. **Cite sources properly** according to the citation anchor format, utilizing group anchors when appropriate.
290+
5. **Tailor your approach to the query type** (academic, news, coding, etc.) while maintaining an expert, journalistic, unbiased tone.
291+
6. **Provide comprehensive information** with specific details, examples, and as much relevant context as possible from search results.
292+
7. **Avoid moralizing language.**
293+
`.trim();
294+
return createSearchTool({
295+
...result.authResult,
296+
onSearchResults,
297+
onGetHighlights,
298+
logger,
299+
});
300+
};
301+
continue;
266302
} else if (tool && appTools[tool] && mcpToolPattern.test(tool)) {
267303
requestedTools[tool] = async () =>
268304
createMCPTool({

api/models/Agent.js

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,12 +60,17 @@ const loadEphemeralAgent = ({ req, agent_id, endpoint, model_parameters: _m }) =
6060
const { model, ...model_parameters } = _m;
6161
/** @type {Record<string, FunctionTool>} */
6262
const availableTools = req.app.locals.availableTools;
63-
const mcpServers = new Set(req.body.ephemeralAgent?.mcp);
63+
/** @type {TEphemeralAgent | null} */
64+
const ephemeralAgent = req.body.ephemeralAgent;
65+
const mcpServers = new Set(ephemeralAgent?.mcp);
6466
/** @type {string[]} */
6567
const tools = [];
66-
if (req.body.ephemeralAgent?.execute_code === true) {
68+
if (ephemeralAgent?.execute_code === true) {
6769
tools.push(Tools.execute_code);
6870
}
71+
if (ephemeralAgent?.web_search === true) {
72+
tools.push(Tools.web_search);
73+
}
6974

7075
if (mcpServers.size > 0) {
7176
for (const toolName of Object.keys(availableTools)) {

api/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@
4848
"@langchain/google-genai": "^0.2.8",
4949
"@langchain/google-vertexai": "^0.2.8",
5050
"@langchain/textsplitters": "^0.1.0",
51-
"@librechat/agents": "^2.4.317",
51+
"@librechat/agents": "^2.4.35",
5252
"@librechat/data-schemas": "*",
5353
"@waylaidwanderer/fetch-event-source": "^3.0.1",
5454
"axios": "^1.8.2",

api/server/controllers/UserController.js

Lines changed: 50 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,10 @@
1-
const { FileSources } = require('librechat-data-provider');
1+
const {
2+
Tools,
3+
Constants,
4+
FileSources,
5+
webSearchKeys,
6+
extractWebSearchEnvVars,
7+
} = require('librechat-data-provider');
28
const {
39
Balance,
410
getFiles,
@@ -83,7 +89,6 @@ const deleteUserFiles = async (req) => {
8389
const updateUserPluginsController = async (req, res) => {
8490
const { user } = req;
8591
const { pluginKey, action, auth, isEntityTool } = req.body;
86-
let authService;
8792
try {
8893
if (!isEntityTool) {
8994
const userPluginsService = await updateUserPluginsService(user, pluginKey, action);
@@ -95,32 +100,55 @@ const updateUserPluginsController = async (req, res) => {
95100
}
96101
}
97102

98-
if (auth) {
99-
const keys = Object.keys(auth);
100-
const values = Object.values(auth);
101-
if (action === 'install' && keys.length > 0) {
102-
for (let i = 0; i < keys.length; i++) {
103-
authService = await updateUserPluginAuth(user.id, keys[i], pluginKey, values[i]);
104-
if (authService instanceof Error) {
105-
logger.error('[authService]', authService);
106-
const { status, message } = authService;
107-
res.status(status).send({ message });
108-
}
103+
if (auth == null) {
104+
return res.status(200).send();
105+
}
106+
107+
let keys = Object.keys(auth);
108+
if (keys.length === 0 && pluginKey !== Tools.web_search) {
109+
return res.status(200).send();
110+
}
111+
const values = Object.values(auth);
112+
113+
/** @type {number} */
114+
let status = 200;
115+
/** @type {string} */
116+
let message;
117+
/** @type {IPluginAuth | Error} */
118+
let authService;
119+
120+
if (pluginKey === Tools.web_search) {
121+
/** @type {TCustomConfig['webSearch']} */
122+
const webSearchConfig = req.app.locals?.webSearch;
123+
keys = extractWebSearchEnvVars({
124+
keys: action === 'install' ? keys : webSearchKeys,
125+
config: webSearchConfig,
126+
});
127+
}
128+
129+
if (action === 'install') {
130+
for (let i = 0; i < keys.length; i++) {
131+
authService = await updateUserPluginAuth(user.id, keys[i], pluginKey, values[i]);
132+
if (authService instanceof Error) {
133+
logger.error('[authService]', authService);
134+
({ status, message } = authService);
109135
}
110136
}
111-
if (action === 'uninstall' && keys.length > 0) {
112-
for (let i = 0; i < keys.length; i++) {
113-
authService = await deleteUserPluginAuth(user.id, keys[i]);
114-
if (authService instanceof Error) {
115-
logger.error('[authService]', authService);
116-
const { status, message } = authService;
117-
res.status(status).send({ message });
118-
}
137+
} else if (action === 'uninstall') {
138+
for (let i = 0; i < keys.length; i++) {
139+
authService = await deleteUserPluginAuth(user.id, keys[i]);
140+
if (authService instanceof Error) {
141+
logger.error('[authService]', authService);
142+
({ status, message } = authService);
119143
}
120144
}
121145
}
122146

123-
res.status(200).send();
147+
if (status === 200) {
148+
return res.status(status).send();
149+
}
150+
151+
res.status(status).send({ message });
124152
} catch (err) {
125153
logger.error('[updateUserPluginsController]', err);
126154
return res.status(500).json({ message: 'Something went wrong.' });

api/server/controllers/agents/callbacks.js

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -237,6 +237,30 @@ function createToolEndCallback({ req, res, artifactPromises }) {
237237
return;
238238
}
239239

240+
if (output.artifact[Tools.web_search]) {
241+
artifactPromises.push(
242+
(async () => {
243+
const name = `${output.name}_${output.tool_call_id}_${nanoid()}`;
244+
const attachment = {
245+
name,
246+
type: Tools.web_search,
247+
messageId: metadata.run_id,
248+
toolCallId: output.tool_call_id,
249+
conversationId: metadata.thread_id,
250+
[Tools.web_search]: { ...output.artifact[Tools.web_search] },
251+
};
252+
if (!res.headersSent) {
253+
return attachment;
254+
}
255+
res.write(`event: attachment\ndata: ${JSON.stringify(attachment)}\n\n`);
256+
return attachment;
257+
})().catch((error) => {
258+
logger.error('Error processing artifact content:', error);
259+
return null;
260+
}),
261+
);
262+
}
263+
240264
if (output.artifact.content) {
241265
/** @type {FormattedContent[]} */
242266
const content = output.artifact.content;

api/server/controllers/agents/client.js

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -39,9 +39,6 @@ const BaseClient = require('~/app/clients/BaseClient');
3939
const { logger, sendEvent } = require('~/config');
4040
const { createRun } = require('./run');
4141

42-
/** @typedef {import('@librechat/agents').MessageContentComplex} MessageContentComplex */
43-
/** @typedef {import('@langchain/core/runnables').RunnableConfig} RunnableConfig */
44-
4542
/**
4643
* @param {ServerRequest} req
4744
* @param {Agent} agent
@@ -543,7 +540,7 @@ class AgentClient extends BaseClient {
543540
}
544541

545542
async chatCompletion({ payload, abortController = null }) {
546-
/** @type {Partial<RunnableConfig> & { version: 'v1' | 'v2'; run_id?: string; streamMode: string }} */
543+
/** @type {Partial<GraphRunnableConfig>} */
547544
let config;
548545
/** @type {ReturnType<createRun>} */
549546
let run;

api/server/controllers/tools.js

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ const {
66
Permissions,
77
ToolCallTypes,
88
PermissionTypes,
9+
loadWebSearchAuth,
910
} = require('librechat-data-provider');
1011
const { processFileURL, uploadImageBuffer } = require('~/server/services/Files/process');
1112
const { processCodeOutput } = require('~/server/services/Files/Code/process');
@@ -24,6 +25,36 @@ const toolAccessPermType = {
2425
[Tools.execute_code]: PermissionTypes.RUN_CODE,
2526
};
2627

28+
/**
29+
* Verifies web search authentication, ensuring each category has at least
30+
* one fully authenticated service.
31+
*
32+
* @param {ServerRequest} req - The request object
33+
* @param {ServerResponse} res - The response object
34+
* @returns {Promise<void>} A promise that resolves when the function has completed
35+
*/
36+
const verifyWebSearchAuth = async (req, res) => {
37+
try {
38+
const userId = req.user.id;
39+
/** @type {TCustomConfig['webSearch']} */
40+
const webSearchConfig = req.app.locals?.webSearch || {};
41+
const result = await loadWebSearchAuth({
42+
userId,
43+
loadAuthValues,
44+
webSearchConfig,
45+
throwError: false,
46+
});
47+
48+
return res.status(200).json({
49+
authenticated: result.authenticated,
50+
authTypes: result.authTypes,
51+
});
52+
} catch (error) {
53+
console.error('Error in verifyWebSearchAuth:', error);
54+
return res.status(500).json({ message: error.message });
55+
}
56+
};
57+
2758
/**
2859
* @param {ServerRequest} req - The request object, containing information about the HTTP request.
2960
* @param {ServerResponse} res - The response object, used to send back the desired HTTP response.
@@ -32,6 +63,9 @@ const toolAccessPermType = {
3263
const verifyToolAuth = async (req, res) => {
3364
try {
3465
const { toolId } = req.params;
66+
if (toolId === Tools.web_search) {
67+
return await verifyWebSearchAuth(req, res);
68+
}
3569
const authFields = fieldsMap[toolId];
3670
if (!authFields) {
3771
res.status(404).json({ message: 'Tool not found' });

api/server/routes/config.js

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,26 @@ router.get('/', async function (req, res) {
8585
bundlerURL: process.env.SANDPACK_BUNDLER_URL,
8686
staticBundlerURL: process.env.SANDPACK_STATIC_BUNDLER_URL,
8787
};
88+
/** @type {TCustomConfig['webSearch']} */
89+
const webSearchConfig = req.app.locals.webSearch;
90+
if (
91+
webSearchConfig != null &&
92+
(webSearchConfig.searchProvider ||
93+
webSearchConfig.scraperType ||
94+
webSearchConfig.rerankerType)
95+
) {
96+
payload.webSearch = {};
97+
}
98+
99+
if (webSearchConfig?.searchProvider) {
100+
payload.webSearch.searchProvider = webSearchConfig.searchProvider;
101+
}
102+
if (webSearchConfig?.scraperType) {
103+
payload.webSearch.scraperType = webSearchConfig.scraperType;
104+
}
105+
if (webSearchConfig?.rerankerType) {
106+
payload.webSearch.rerankerType = webSearchConfig.rerankerType;
107+
}
88108

89109
if (ldap) {
90110
payload.ldap = ldap;

api/server/services/AppService.js

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
const {
22
FileSources,
3-
EModelEndpoint,
43
loadOCRConfig,
54
processMCPEnv,
5+
EModelEndpoint,
66
getConfigDefaults,
7+
loadWebSearchConfig,
78
} = require('librechat-data-provider');
89
const { checkVariables, checkHealth, checkConfig, checkAzureVariables } = require('./start/checks');
910
const { azureAssistantsDefaults, assistantsConfigSetup } = require('./start/assistants');
@@ -35,6 +36,7 @@ const AppService = async (app) => {
3536
const configDefaults = getConfigDefaults();
3637

3738
const ocr = loadOCRConfig(config.ocr);
39+
const webSearch = loadWebSearchConfig(config.webSearch);
3840
const filteredTools = config.filteredTools;
3941
const includedTools = config.includedTools;
4042
const fileStrategy = config.fileStrategy ?? configDefaults.fileStrategy;
@@ -79,6 +81,7 @@ const AppService = async (app) => {
7981
const defaultLocals = {
8082
ocr,
8183
paths,
84+
webSearch,
8285
fileStrategy,
8386
socialLogins,
8487
filteredTools,

api/server/services/AppService.spec.js

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,14 @@ describe('AppService', () => {
141141
balance: { enabled: true },
142142
filteredTools: undefined,
143143
includedTools: undefined,
144+
webSearch: {
145+
cohereApiKey: '${COHERE_API_KEY}',
146+
firecrawlApiKey: '${FIRECRAWL_API_KEY}',
147+
firecrawlApiUrl: '${FIRECRAWL_API_URL}',
148+
jinaApiKey: '${JINA_API_KEY}',
149+
safeSearch: true,
150+
serperApiKey: '${SERPER_API_KEY}',
151+
},
144152
});
145153
});
146154

@@ -537,7 +545,7 @@ describe('AppService updating app.locals and issuing warnings', () => {
537545
const { logger } = require('~/config');
538546
expect(logger.warn).toHaveBeenCalledWith(
539547
expect.stringContaining(
540-
'The \'assistants\' endpoint has both \'supportedIds\' and \'excludedIds\' defined.',
548+
"The 'assistants' endpoint has both 'supportedIds' and 'excludedIds' defined.",
541549
),
542550
);
543551
});
@@ -559,7 +567,7 @@ describe('AppService updating app.locals and issuing warnings', () => {
559567
const { logger } = require('~/config');
560568
expect(logger.warn).toHaveBeenCalledWith(
561569
expect.stringContaining(
562-
'The \'assistants\' endpoint has both \'privateAssistants\' and \'supportedIds\' or \'excludedIds\' defined.',
570+
"The 'assistants' endpoint has both 'privateAssistants' and 'supportedIds' or 'excludedIds' defined.",
563571
),
564572
);
565573
});

0 commit comments

Comments
 (0)