postgres/typesense
1
0
Fork 0
mirror of https://github.com/typesense/typesense.git synced 2025-05-19 05:08:43 +08:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Merge pull request #1898 from joonatanaatos/main

Browse Source 
Do housekeeping for authmanager at regular intervals
This commit is contained in:
Kishore Nallan 2024-08-16 10:21:17 +05:30 committed by GitHub
commit 673d36f2b0
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 104 additions and 65 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
README.md
View File

@ -14,13 +14,13 @@
<!-- <a href="https://circleci.com/gh/typesense/typesense"><img src="https://circleci.com/gh/typesense/typesense.svg?style=shield&circle-token=1addd775339738a3d90869ddd8201110d561feaa"></a> -->
<a href="https://hub.docker.com/r/typesense/typesense/tags"><img src="https://img.shields.io/docker/pulls/typesense/typesense"></a>
<a href="https://github.com/typesense"><img src="https://img.shields.io/github/stars/typesense/typesense?label=github%20stars&style=flat"></a><br>
<a href="https://cloud.typesense.org"><img src="https://img.shields.io/badge/searches_per_month_on_typesense_cloud-3 Billion-blue"></a>
<a href="https://cloud.typesense.org"><img src="https://img.shields.io/badge/searches_per_month_on_typesense_cloud-10 Billion-blue"></a>
<p>
<p align="center">
<a href="https://typesense.org">Website</a> |
<a href="https://typesense.org/docs/">Documentation</a> |
<a href="https://github.com/orgs/typesense/projects/1">Roadmap</a> |
<a href="https://join.slack.com/t/typesense-community/shared_invite/zt-2fetvh0pw-ft5y2YQlq4l_bPhhqpjXig">Slack Community</a> |
<a href="https://join.slack.com/t/typesense-community/shared_invite/zt-2otyo41xs-tbZNeeC6F37_FKftAdUc5A">Slack Community</a> |
<a href="https://threads.typesense.org/kb">Community Threads</a> |
<a href="https://twitter.com/typesense">Twitter</a>
</p>
@ -115,7 +115,7 @@ We'd love to benchmark with larger datasets, if we can find large ones in the pu
Typesense is used by a range of users across different domains and verticals.
On Typesense Cloud we serve more than **3 BILLION** searches per month. Typesense's Docker images have been downloaded over 12M times.
On Typesense Cloud we serve more than **10 BILLION** searches per month. Typesense's Docker images have been downloaded over 12M times.
We've recently started documenting who's using it in our [Showcase](SHOWCASE.md).
If you'd like to be included in the list, please feel free to edit [SHOWCASE.md](SHOWCASE.md) and send us a PR.
@ -351,7 +351,7 @@ If you have specifics that prevent you from using Typesense due to a licensing i
## Support
👋 🌐 If you have general questions about Typesense, want to say hello or just follow along, we'd like to invite you to join our public [Slack Community](https://join.slack.com/t/typesense-community/shared_invite/zt-2fetvh0pw-ft5y2YQlq4l_bPhhqpjXig).
👋 🌐 If you have general questions about Typesense, want to say hello or just follow along, we'd like to invite you to join our public [Slack Community](https://join.slack.com/t/typesense-community/shared_invite/zt-2otyo41xs-tbZNeeC6F37_FKftAdUc5A).
If you run into any problems or issues, please create a GitHub issue and we'll try our best to help.
@ -375,7 +375,7 @@ If you'd like to get updates when we release new versions, click on the "Watch"
We also post updates to our Twitter account about releases and additional topics related to Typesense. Follow us here: [@typesense](https://twitter.com/typesense).
👋 🌐 We'll also post updates on our [Slack Community](https://join.slack.com/t/typesense-community/shared_invite/zt-2fetvh0pw-ft5y2YQlq4l_bPhhqpjXig).
👋 🌐 We'll also post updates on our [Slack Community](https://join.slack.com/t/typesense-community/shared_invite/zt-2otyo41xs-tbZNeeC6F37_FKftAdUc5A).
## Build from source

144
SHOWCASE.md
View File

@ -2,65 +2,95 @@
This is a selection of sites and apps using Typesense.
> As of Mar 2022 we have around 2.7M Docker pulls. So the list below is a tiny sample of everyone using Typesense.
>
> As of Jul 2024 we have around 10M+ Docker pulls. So the list below is a tiny sample of everyone using Typesense.
>
> If you are a Typesense user, an easy way to contribute back to the ecosystem is by sharing that you are indeed using Typesense. This helps build confidence among potential new users who are exploring Typesense.
> So we invite and welcome you to add your own sites / apps to the list above. Please edit this file and send us a PR 🙏
> So we invite and welcome you to add your own sites / apps to the list above. Please edit this file and send us a PR 🙏
| Name | Description |
| ----------- | ----------- |
| [Aviyel](https://aviyel.com) | Open-source community platform |
| [Back Right Up](https://backrightup.com) | Automatic Azure Devops backups |
| [BetaSeries](https://www.betaseries.com/) | TV shows and movies recommendation, using Typesense to search into 4M+ international records |
| [Bikifi](https://bikifi.com/) | A free education platform that brings premium educational content to anyone, anywhere in Turkey |
| [Bookology](https://play.google.com/store/apps/details?id=com.imihirpaldhikar.bookology) | "I am using Typesense for searching the the books listed on the platform. I switched from Algolia to Typesense because it is more convenient to only pay for the underlying infrastructure rather then paying for search queries. I will always use Typesense in my future projects." - @imihirpaldhikar
| [Cults.](https://cults3d.com) | Find designs for 3D-printing. *We are very happy with our switch to Typesense, making it a breeze to setup and have pertinent results right from the start* |
| [datarequests.org](https://www.datarequests.org/) | Search backend for the [company database](https://www.datarequests.org/company) that allows users to easily find the contact details for privacy-related requests |
| [Dave & Adams Card World](https://www.dacardworld.com) | E-commerce platform for collectibles like sports cards, gaming cards, comics, or autographed memorabilia |
| [DiscMaster](http://discmaster.textfiles.com/search) | Search 94 million vintage computer files from archive.org |
| [Echidna Sewing](https://www.echidnasewing.com.au) | Echidna Sewing is one of Australia’s largest sewing machine dealers |
| [ElbiseBul](https://www.elbisebul.com/) | E-commerce |
| [FlyGRN](https://flygrn.com) | Flight Ticket Search Engine that ranks flights on carbon emissions and gives greener alternatives |
| [furn.nl](https://furn.nl) | Dutch furniture comparison website |
| [GlobalTill](https://www.globaltill.com) | Omnichannel Retail Management System |
| [Grafikart](https://www.grafikart.fr/) | Learning resources library |
| [Granthappura](https://gpura.org/) | A free-licensed digitized artifacts archive related to Kerala from multiple languages. [Github Link](https://github.com/indic-archive/omeka-typesense-search) |
| [Have A Class](https://haveaclass.com/) | Find the perfect teacher online |
| [Hem](https://hem.com) | A premium furniture brand for the auction houses of tomorrow |
| [Iconduck](https://iconduck.com) | Site to search for open source icons |
| [Instatus](https://instatus.com) | System Status Pages Platform |
| [Job Ads Now](https://www.jobadsnow.com) | Job Ads Now is a jobs search engine focusing on the best candidate experience. Blazing fast search and filters by salary, location, bonus and more. You can also read the job highlights (TL;DR) for each job identified from the origina job post using AI |
| [Jobsort](https://www.jobsort.com/) | Job search engine for developers, by developers (quicksort for tech jobs.) |
| [Lawyaw](https://lawyaw.com) | Lawyaw provides easy to use document automation software that helps legal practices save time and avoid mistakes without sacrificing quality. *"Typesense is used as our search index and works great to search, sort and facet thousands of our Court forms each day. We look forward to working with Typesense for years to come as it continues to mature!" - @tpayne84* |
| [Magny SaaS search bar](https://magny.io) | Magny is a SaaS platform that helps implement command palettes, which significantly decreases the time users find anything in your app, leveraging the user experience and efficiency. |
| [MyHearty](https://myhearty.my) | A one-stop charity website to fundraise, donate, volunteer and apply for aids. It comes with a full-featured search UI (instant search, geosearch) for campaigns, volunteer events and aids. |
| [NeoDB](https://neodb.social/) | An open source project and free service to help users in Fediverse manage, share and discover collections, reviews and ratings for culture products, with a searchable catalog of over a million books, movies, music, podcasts, games and performances. |
| [Neolace](https://www.neolace.com) | Knowledge graph platform (combines a wiki and a graph database) |
| [New York University Databrary](https://nyu.databrary.org/) | Video documentation library |
| [Oddle Eats](https://eats.oddle.me) | Food Delivery Service in Singapore |
| [OpenTacos](https://tacos.openbeta.io) | Free and open source rock climbing route wiki |
| [Orders Made Simple](https://ordersmadesimple.com) | A Stock ordering platform helping to connect hospitality and catering venues with their suppliers |
| [Otaku Square](https://www.otakusquare.com) | Online retailer for pop-culture (e.g. anime, games, movies) merchandise |
| [Plaza](https://www.useplaza.com) | Live-shopping platform |
| [Popcorn](https://onpopcorn.com) | Zoom meets Snapchat, async video chat app |
| [Project Seldon](https://projectseldon.org) | A free and unbiased analysis platform for understanding societies and systems of government. |
| [Puey Ungphakorn Institute for Economic Research](https://www.pier.or.th) | A research institute in Thailand |
| [Read This Twice](https://www.readthistwice.com/) | Book discovery platform uses typesense to power the book/people search |
| [Rushtix](https://rushtix.com) | Live Comedy Shows platform |
| [Service Foods](https://www.servicefoods.co.nz) | One of New Zealand's largest privately family owned and operated food service distribution businesses |
| [Syracuse University](https://syracuse.edu) | A private research university in Syracuse, New York. [Visit our live typesense demo!](https://typesense-demo.dev3.digitalservices.syr.edu) |
| [Trading Strategy](https://tradingstrategy.ai/blog/world-fastest-token-search) | Algorithmic trading protocol for decentralised markets |
| [Upnext](https://www.getupnext.com) | Save, organize and focus on your personal content library |
| [What's In My Jar](https://whatsinmyjar.com) | Skincare Research platform |
| Name | Description |
| ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | ---------------------------------- | --------------------------------------------------------------- |
| [AEROTRADE](https://ade.aero) | Digital marketplace for aircraft parts |
| [Agriland](https://agriland.ie) | Leading news platform focused on farming and agricultural topics in Ireland |
| [Aviyel](https://aviyel.com) | Open-source community platform |
| [Back Right Up](https://backrightup.com) | Automatic Azure Devops backups |
| [Beatopia](https://www.beatopia.com/) | Subscription-based platform for rappers and vocalists to access high-quality beats produced by industry professionals |
| [BetaSeries](https://www.betaseries.com/) | TV shows and movies recommendation, using Typesense to search into 4M+ international records |
| [Bikifi](https://bikifi.com/) | A free education platform that brings premium educational content to anyone, anywhere in Turkey |
| [Bookology](https://play.google.com/store/apps/details?id=com.imihirpaldhikar.bookology) | "I am using Typesense for searching the the books listed on the platform. I switched from Algolia to Typesense because it is more convenient to only pay for the underlying infrastructure rather then paying for search queries. I will always use Typesense in my future projects." - @imihirpaldhikar |
| [Changelog](https://changelog.com) | Developer-centric podcast platform |
| [Codecademy](https://www.codecademy.com/) | Online learning platform for computer science |
| [Cults.](https://cults3d.com) | Find designs for 3D-printing. _We are very happy with our switch to Typesense, making it a breeze to setup and have pertinent results right from the start_ |
| [datarequests.org](https://www.datarequests.org/) | Search backend for the [company database](https://www.datarequests.org/company) that allows users to easily find the contact details for privacy-related requests |
| [Dave & Adams Card World](https://www.dacardworld.com) | E-commerce platform for collectibles like sports cards, gaming cards, comics, or autographed memorabilia |
| [DiscMaster](http://discmaster.textfiles.com/search) | Search 94 million vintage computer files from archive.org |
| [Echidna Sewing](https://www.echidnasewing.com.au) | Echidna Sewing is one of Australia’s largest sewing machine dealers |
| [ElbiseBul](https://www.elbisebul.com/) | E-commerce |
| [FlyGRN](https://flygrn.com) | Flight Ticket Search Engine that ranks flights on carbon emissions and gives greener alternatives |
| [Foodnoms](https://foodnoms.com) | Food and nutrition tracking app |
| [furn.nl](https://furn.nl) | Dutch furniture comparison website |
| [GlobalTill](https://www.globaltill.com) | Omnichannel Retail Management System |
| [Grafikart](https://www.grafikart.fr/) | Learning resources library |
| [Granthappura](https://gpura.org/) | A free-licensed digitized artifacts archive related to Kerala from multiple languages. [Github Link](https://github.com/indic-archive/omeka-typesense-search) |
| [Hardcover](https://hardcover.app) | A social network app designed for book lovers to discover, track, and share books. |
| [Have A Class](https://haveaclass.com/) | Find the perfect teacher online |
| [Hem](https://hem.com) | A premium furniture brand for the auction houses of tomorrow |
| [Iconduck](https://iconduck.com) | Site to search for open source icons |
| [Instatus](https://instatus.com) | System Status Pages Platform |
| [Kick](http://kick.com/) | Livestreaming Platform |
| [KickstandsUp](http://kickstandsup.com/) | Find Motorcycle events near you |
| [Job Ads Now](https://www.jobadsnow.com) | Job Ads Now is a jobs search engine focusing on the best candidate experience. Blazing fast search and filters by salary, location, bonus and more. You can also read the job highlights (TL;DR) for each job identified from the origina job post using AI |
| [Jobsort](https://www.jobsort.com/) | Job search engine for developers, by developers (quicksort for tech jobs.) |
| [Lawyaw](https://lawyaw.com) | Lawyaw provides easy to use document automation software that helps legal practices save time and avoid mistakes without sacrificing quality. _"Typesense is used as our search index and works great to search, sort and facet thousands of our Court forms each day. We look forward to working with Typesense for years to come as it continues to mature!" - @tpayne84_ |
| [Logitech](https://www.logitech.com/) | Computer/technology peripherals |
| [Lonely Planet](https://www.lonelyplanet.com/) | Travel Guide Platform |
| [Macro Factor](https://macrofactorapp.com) | Nutrition and diet tracking application designed to help users achieve their fitness and health goals |
| [Magny SaaS search bar](https://magny.io) | Magny is a SaaS platform that helps implement command palettes, which significantly decreases the time users find anything in your app, leveraging the user experience and efficiency. |
| [MyHearty](https://myhearty.my) | A one-stop charity website to fundraise, donate, volunteer and apply for aids. It comes with a full-featured search UI (instant search, geosearch) for campaigns, volunteer events and aids. |
| [n8n](https://n8n.io/) | Free and source-available fair-code licensed workflow automation tool |
| [NeoDB](https://neodb.social/) | An open source project and free service to help users in Fediverse manage, share and discover collections, reviews and ratings for culture products, with a searchable catalog of over a million books, movies, music, podcasts, games and performances. | | [Neolace](https://www.neolace.com) | Knowledge graph platform (combines a wiki and a graph database) |
| [Neolace](https://www.neolace.com) | Knowledge graph platform (combines a wiki and a graph database) |
| [New York University Databrary](https://nyu.databrary.org/) | Video documentation library |
| [Oddle Eats](https://eats.oddle.me) | Food Delivery Service in Singapore |
| [OpenTacos](https://tacos.openbeta.io) | Free and open source rock climbing route wiki |
| [Orders Made Simple](https://ordersmadesimple.com) | A Stock ordering platform helping to connect hospitality and catering venues with their suppliers |
| [Otaku Square](https://www.otakusquare.com) | Online retailer for pop-culture (e.g. anime, games, movies) merchandise |
| [Plaza](https://www.useplaza.com) | Live-shopping platform |
| [Pangobooks](https://www.pangobooks.com) | Online marketplace for buying and selling used books |
| [Poorvika](https://poorvika.com) | Indian technology online retailer |
| [Popcorn](https://onpopcorn.com) | Zoom meets Snapchat, async video chat app |
| [Project Seldon](https://projectseldon.org) | A free and unbiased analysis platform for understanding societies and systems of government. |
| [Puey Ungphakorn Institute for Economic Research](https://www.pier.or.th) | A research institute in Thailand |
| [Read This Twice](https://www.readthistwice.com/) | Book discovery platform uses typesense to power the book/people search |
| [Reelgood](https://reelgood.com/) | A comprehensive streaming guide that helps users find and track TV shows and movies across a wide range of streaming services |
| [Rushtix](https://rushtix.com) | Live Comedy Shows platform |
| [Service Foods](https://www.servicefoods.co.nz) | One of New Zealand's largest privately family owned and operated food service distribution businesses |
| [Solid Starts](https://solidstarts.com/) | A comprehensive resource designed to help parents introduce solid foods to their babies |
| [Soundsnap](https://soundsnap.com/) | Sound effects library |
| [Squire](https://getsquire.com/) | Barbershop management system |
| [Syracuse University](https://syracuse.edu) | A private research university in Syracuse, New York. [Visit our live typesense demo!](https://typesense-demo.dev3.digitalservices.syr.edu) |
| [TackleTarts](https://tackletarts.uk/) | An online store specializing in fishing tackle and equipment in the UK |
| [Trading Strategy](https://tradingstrategy.ai/blog/world-fastest-token-search) | Algorithmic trading protocol for decentralised markets |
| [Ultrahuman](https://www.ultrahuman.com) | Health and wellness platform that focuses on metabolic fitness and real-time health tracking through advanced wearable technology |
| [Upnext](https://www.getupnext.com) | Save, organize and focus on your personal content library |
| [What's In My Jar](https://whatsinmyjar.com) | Skincare Research platform |
### Typesense Demos
| Name | Description |
| ----------- | ----------- |
| [Recipe Search](https://recipe-search.typesense.org/) | A site that showcases Typesense in action on a 2 Million recipe database, with the ability to filter by ingredients.|
| [Linux Commit History Search](https://linux-commits-search.typesense.org/) | A site that indexes 1M linux commit messages in Typesense and lets you browse, search and filter through the commits.|
| [Books Search](https://books-search.typesense.org/) | A site that showcases Typesense in action on a 28 Million books database from [OpenLibrary](https://openlibrary.org/), with the ability to filter by authors and subject. |
| [MusicBrainz Songs Search](https://songs-search.typesense.org/) | A site that showcases Typesense in action on a 32 Million Songs database from [MusicBrainz](https://musicbrainz.org/) |
| [2020 US Presidential Candidates' Speeches Search](https://biden-trump-speeches-search.typesense.org/) | Instantly Search speeches of US Presidential Candidates side-by-side. |
| [E-commerce Store Browsing Experience](https://ecommerce-store.typesense.org/) | A site that showcases how to build an e-commerce storefront browsing experience with Typesense. |
| Name | Description |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Recipe Search](https://recipe-search.typesense.org/) | A site that showcases Typesense in action on a 2 Million recipe database, with the ability to filter by ingredients. |
| [Linux Commit History Search](https://linux-commits-search.typesense.org/) | A site that indexes 1M linux commit messages in Typesense and lets you browse, search and filter through the commits. |
| [E-commerce Store Browsing Experience](https://ecommerce-store.typesense.org/) | A site that showcases how to build an e-commerce storefront browsing experience with Typesense. |
| [E-commerce Storefront with Next.js + Typesense + Vercel](https://showcase-nextjs-typesense-ecommerce-store.vercel.app) | A site that showcases how to build an e-commerce storefront browsing experience with Typesense and Next.js. |
| [xkcd comic search](https://findxkcd.com/) | A site that lets you search xkcd comics by topic, with data indexed in Typesense. |
| [Books Search](https://books-search.typesense.org/) | A site that showcases Typesense in action on a 28 Million books database from [OpenLibrary](https://openlibrary.org/), with the ability to filter by authors and subject. |
| [MusicBrainz Songs Search](https://songs-search.typesense.org/) | A site that showcases Typesense in action on a 32 Million Songs database from [MusicBrainz](https://musicbrainz.org/) |
| [Typeahead Spellchecker ](https://spellcheck.typesense.org/) | A widget that mimics the typeahead spellcheckers on Android / iPhone keyboards. |
| [2020 US Presidential Candidates' Speeches Search](https://biden-trump-speeches-search.typesense.org/) | Instantly Search speeches of US Presidential Candidates side-by-side. |
| [Federated Search](https://federated-search.typesense.org/) | A site that showcases Typesense's Federated Search feature. |
| [Federated Search](https://federated-search.typesense.org/) | A site that showcases Typesense's Federated Search feature. |
| [AirBnB Geo Search](https://airbnb-geosearch.typesense.org/) | A site that showcases Typesense's Geo Search feature. |
| [Airports Geo Search](https://airports-geosearch.typesense.org/) | A site that showcases Typesense's Geo Search feature with Next.js and react-instantsearch. |
| [Semantic Search + Hybrid Search](https://hn-comments-search.typesense.org/) | A site that showcases Typesense's semantic search and hybrid search features, along with the auto-embedding generation feature. |
| Guitar Chords Search in [Nuxtjs](https://guitar-chords-search-nuxt-js.typesense.org/), [Nextjs](https://guitar-chords-search-next-js.typesense.org/), [Angular](https://guitar-chords-search-angular.typesense.org/), [Vanilla JS](https://guitar-chords-search-vanilla-js.typesense.org/), [Astro](https://guitar-chords-search-astro.typesense.org/), [SolidJS](https://guitar-chords-search-solid-js.typesense.org/), [Remix](https://guitar-chords-search-remix.typesense.org/), [SvelteKit](https://guitar-chords-search-svelte-kit.typesense.org/), [Qwik](https://guitar-chords-search-qwik.typesense.org/), and [React Native](https://github.com/typesense/showcase-guitar-chords-search-react-native) | Sites that showcase how to use Typesense in different Javascript frameworks. |
| [Address autocomplete](https://address-autocomplete.typesense.org/) | An address autocomplete experience powered by Typesense. |

1
include/housekeeper.h
View File

@ -10,6 +10,7 @@ private:
std::atomic<bool> quit = false;
std::atomic<uint32_t> hnsw_repair_interval_s = 1800;
std::atomic<uint32_t> remove_expired_keys_interval_s = 3600;
HouseKeeper() {}

14
src/housekeeper.cpp
View File

@ -5,6 +5,9 @@ void HouseKeeper::run() {
uint64_t prev_hnsw_repair_s = std::chrono::duration_cast<std::chrono::seconds>(
std::chrono::system_clock::now().time_since_epoch()).count();
uint64_t prev_remove_expired_keys_s = std::chrono::duration_cast<std::chrono::seconds>(
std::chrono::system_clock::now().time_since_epoch()).count();
uint64_t prev_db_compaction_s = std::chrono::duration_cast<std::chrono::seconds>(
std::chrono::system_clock::now().time_since_epoch()).count();
@ -48,13 +51,18 @@ void HouseKeeper::run() {
LOG(INFO) << "Ran housekeeping for " << coll_names.size() << " collections.";
}
//do housekeeping for authmanager
CollectionManager::get_instance().getAuthManager().do_housekeeping();
prev_hnsw_repair_s = std::chrono::duration_cast<std::chrono::seconds>(
std::chrono::system_clock::now().time_since_epoch()).count();
}*/
if (now_ts_seconds - prev_remove_expired_keys_s >= remove_expired_keys_interval_s) {
// Do housekeeping for authmanager
CollectionManager::get_instance().getAuthManager().do_housekeeping();
prev_remove_expired_keys_s = std::chrono::duration_cast<std::chrono::seconds>(
std::chrono::system_clock::now().time_since_epoch()).count();
}
lk.unlock();
}
}