Arindam Majumder

Posted on Nov 14 • Edited on Nov 18

9 Open Source Libraries to Supercharge Your Next Project 🔋⚡️

#opensource #webdev #javascript #beginners

As a developer, we often get confused about which tools to use to build our Next (side) project.

Even if you’re serious enough to build a product, the problem remains the same.

So, I’m covering 9 Open Source Libraries that will truly supercharge your future project!

Let’s go 🚀

1. Crawlee - Web Scraping, Simplified and Powerful

Crawlee is a complete web scraping and browser automation library that helps us build reliable crawlers quickly and efficiently.

With its built-in anti-blocking features, you can build human like bots which will reduce the chances of getting blocked.

Whether you're using Node.js or Python, Crawlee gives a unified interface for HTTP and headless browser crawling, making it adaptable for all kinds of scraping tasks.

You can watch the introduction video to know more about it:

The easiest way to start using Crawlee is with the Crawlee CLI. Run the following command:

npx crawlee create my-crawler

This will set up a new project with all the dependencies you need to get started.

If you want to add Crawlee to your project, run the command below:

npm install crawlee playwright

Now, here’s a quick example:

import { PlaywrightCrawler } from 'crawlee';

// PlaywrightCrawler crawls the web using a headless browser controlled by the Playwright library.
const crawler = new PlaywrightCrawler({
    // Use the requestHandler to process each of the crawled pages.
    async requestHandler({ request, page, enqueueLinks, pushData, log }) {
        const title = await page.title();
        log.info(`Title of ${request.loadedUrl} is '${title}'`);

        // Save results as JSON to `./storage/datasets/default` directory.
        await pushData({ title, url: request.loadedUrl });

        // Extract links from the current page and add them to the crawling queue.
        await enqueueLinks();
    },

    // Uncomment this option to see the browser window.
    // headless: false,

    // Comment this option to scrape the full website.
    maxRequestsPerCrawl: 20,
});

// Add first URL to the queue and start the crawl.
await crawler.run(['https://crawlee.dev']);

// Export the whole dataset to a single file in `./result.csv`.
await crawler.exportData('./result.csv');

// Or work with the data directly.
const data = await crawler.getData();
console.table(data.items);

This example, crawls pages, extracts data, and enqueues links. You can learn more from the Docs.

Also, I’ve built a LinkedIn Scrapper using Crawlee for Python, You can give it a read as well:

How to create a LinkedIn job scraper in Python with Crawlee

Arindam Majumder for Crawlee ・ Oct 17

#python #webdev #programming #beginners

They have 15k+ stars on GitHub and 110+ releases, so they are constantly evolving and improving.

Star Crawlee on GitHub ⭐️

2. Encore - Backend framework for type-safe applications

Managing Cloud services for scalable applications is not easy. It often gets messy with complex infrastructure management, inconsistent APIs, and scattered DevOps processes.

Encore simplifies this chaos by providing a unified development platform integrating type-safe backend frameworks, automatic infrastructure provisioning, and DevOps automation.

It is available both in Golang and Typescript. You can see the Intro video to know more:

Using Encore, you can create production-ready backends using primitives like microservices, Postgres, and Pub/Sub, all without the usual complexity and DevOps work.

Get started with Encore by installing the CLI.

curl -L https://encore.dev/install.sh | bash

Then create an app with the encore CLI:

encore app create

This will configure your free account, allow you to choose your app's name, and select the Hello World template.

This will then create an example application with a simple REST API in a new folder using your chosen app name.

Open the file in your editor and go to your-app-name/hello/hello.ts :

import { api } from "encore.dev/api";

export const world = api(
  { method: "GET", path: "/hello/:name", expose: true },
  async ({ name }: { name: string }): Promise<Response> => {
    return { message: `Hello ${name}!` };
  },
);

interface Response {
  message: string;
}

Here, we defined an API endpoint by wrapping a regular async function in a call to api.

Encore automatically handles authentication, HTTP routing, request validation, error handling, observability, API documentation, and more.

For more information, you can refer to their documentation. Also Join their Discord community to stay updated.

Star Encore on GitHub ⭐️

3. Taipy - Build AI web apps in Python

Have you ever wanted to build a simple AI project without complicating it by using a front-end framework?

If so, this tool is for you.

Taipy is an open-source library for building production-ready AI apps faster in Python. It allows you to go from simple pilots to production-ready web applications quickly.

It builds components, graphical interfaces, and dashboards using Python commands. It's primarily used for data-driven projects and is ideal for handling large datasets.

Taipy is well-suited for complex workflows, offering nodes to process them efficiently, and it integrates with pandas, allowing you to manipulate data directly within the project.

You can get started with Taipy pretty quickly, run the following command:

pip install taipy

We can create a GUI using Taipy with the following code:

from taipy.gui import Gui
import taipy.gui.builder as tgb
from math import cos, exp

value = 10

def compute_data(decay:int)->list:
    return [cos(i/6) * exp(-i*decay/600) for i in range(100)]

def slider_moved(state):
    state.data = compute_data(state.value)

with tgb.Page() as page:
    tgb.text(value="# Taipy Getting Started", mode="md")
    tgb.text(value="Value: {value}")
    tgb.slider(value="{value}", on_change=slider_moved)
    tgb.chart(data="{data}")

data = compute_data(value)

if __name__ == "__main__":
    Gui(page=page).run(title="Dynamic chart")

For more details, you can check their Documentation.

With over 16k+ stars on GitHub, it has a strong and growing community, star the project to show your support:

Star Taipy on GitHub ⭐️

4. KitOps - Manage AI Models with Flexibility

KitOps is an open-source tool designed to unify the way data scientists, developers, and SREs manage and collaborate on AI/ML models, source codes, datasets, and other artifacts.

KitOps brings a whole new level of flexibility and efficiency to AI/ML model management, transforming the way teams package, version, and deploy their resources by using OCI standards for maximum compatibility and collaboration.

It comes with a suite of powerful features:

✅ Package models, datasets, configurations, and code as OCI-compliant ModelKits for easy versioning.

✅ Compatible with any OCI-compliant registry for flexibility across AI/ML projects.

✅ YAML-based configuration for fine-tuning LLMs and creating RAG pipelines.

Explore the documentation to discover why KitOps is becoming a must-have in AI/ML workflows!

KitOps also enables flexible management of models, datasets, and code across any OCI-compliant registry, making it ideal for developers and MLOps teams working on both small projects and large-scale enterprise AI applications.

✅ Package and version models, datasets, and code with OCI standards.

✅ Automate model deployment and testing.

✅ Track provenance and secure model assets with artifact signing.

✅ Integrate seamlessly with existing CI/CD workflows.

KitOps is actively evolving with a growing community - join their Discord or star them on GitHub to stay updated and contribute to this exciting project!

Star KitOps on GitHub ⭐️

5. CopilotKit - Build In-App AI Agents Easily

If you’re trying to add AI to your application, CopilotKit is probably the simplest and production-ready way to do so.

CopilotKit is the leading, most robust, and easiest-to-use open-source framework for building in-app AI copilots. You can build in-app AI chatbots, and in-app AI Agents with simple components provided by Copilotkit.

Some of the standout features:

✅ In-App AI Chatbot: Add an AI chatbot to your app with easy plug-and-play components, including support for headless UI, making it incredibly versatile.

✅ Copilot Readable State: This feature allows copilots to access and interpret the application state, making the AI assistant more context-aware and responsive.

✅ Copilot Actions: Enable your copilots to perform tasks and take actions directly within the application based on user inputs and app context.

They also provide built-in (fully customizable) Copilot-native UX components like <CopilotKit />, <CopilotPopup />, <CopilotSidebar />, <CopilotTextarea /> .

If You have ever tried integrating agents in your React application, you know how tricky and time-consuming it can be!

CopilotKit Recently launched Co-Agents to solve this problem.

Co-Agents, a powerful toolset for building Agent-Native applications with LangGraph agents. It is designed to help you deeply integrate LangGraph agents into React applications, pushing the boundaries of what’s possible with AI-native UX.

You can check this video to see what you can do with Co-agents:

I’ve also built an AI-powered Scheduling app using CopilotKit, You can read it here:

🧑‍💻 How I Built the World's Best NextJS, AI Scheduling App 🤖✨

Arindam Majumder for CopilotKit ・ Oct 24

#webdev #ai #javascript #programming

With 12.6k+ stars on GitHub and ongoing updates, CopilotKit has wider community support and trust.

Star CopilotKit on GitHub ⭐️

6. Tolgee - Web-based localization platform

Internationalization and localization are crucial for reaching a global audience, but managing both manually can be very time-consuming.

That’s where Tolgee comes in.

Tolgee is an open-source localization platform that makes it incredibly easy to translate your applications into any language without modifying your code.

It's perfect for web, mobile, and even desktop applications, designed with a developer-first approach to streamline localization from start to finish.

It has a range of powerful features:

✅ In-Context Translation: See translations exactly as they’ll appear in your app.

✅ Manage and Track Translation Keys: Save time by keeping your localization keys in order.

✅ Set Up Roles and Permissions: Control who has access to what for more efficient collaboration.

✅ Extendable with Plugins: Easily integrate Tolgee with additional tools and services as needed.

Tolgee also provides SDKs, a REST API, and CLI to make localization even easier across JavaScript, mobile, and backend environments.

You can join their Slack Community or star them on Github:

Star Tolgee on GitHub ⭐️

7. OPAL - Administration layer for Policy Engine

OPAL is an open-source administration layer designed to work with policy engines like Open Policy Agent (OPA) and AWS Cedar.

If you’re a developer working on large projects with complex access controls, OPAL simplifies the process of managing permissions across teams or user groups.

It detects changes to policies and policy data in real-time, ensuring your app’s authorization layer stays up-to-date.

Whether your data changes via APIs, Git, or third-party services, OPAL automatically syncs the necessary policies and authorization data to your services with proper access flow.

Now, let’s have a look at some of the features and use cases of OPAL:

✅ Real-Time Permission Updates: OPAL automatically grants or revokes permissions in real-time, eliminating the need for manual access updates across applications.

✅ Policy as Code: Developers can write policies as code, making them easier to version, review, and manage for consistent policy handling.

✅ Cloud-Native Integration: OPAL easily integrates with cloud-based microservices like AWS or Kubernetes, streamlining secure and automated policy updates.

✅ Fine-Grained Authorization: OPAL supports precise, end-to-end authorization and is compatible with multiple policy languages and decentralized data sources, making it ideal for complex, distributed systems.

This tool is very helpful for adding real-time permission updates to your project, and it also allows you to easily review and manage user roles at different levels in large applications.

With 4.5k stars on GitHub, it has a strong community, join their Slack or star the project to show your support:

Star OPAL on GitHub ⭐️

8. ChartDB - Visualize Databases Instantly

ChartDB is a powerful, web-based tool that helps developers visualize and manage their database schemas quickly and intuitively.

Whenever I need to understand or design a database, ChartDB is my go-to solution for easy schema visualization.

ChartDB offers a range of essential features:

✅ Instant Schema Import: Run a single "Smart Query" to retrieve your database schema as JSON instantly. This feature saves time, whether you're documenting, collaborating, or gaining insights into your data structure.

✅ AI-Powered Export for Migration: ChartDB’s AI-driven export lets you generate DDL scripts tailored to your target database dialect, making migrations between databases like MySQL, PostgreSQL, SQLite, and more seamless.

✅ Interactive Schema Editing: Easily fine-tune your schema layout in ChartDB's editor. Adjust structures, add annotations, and make visualizations of even the most complex database structures clearer and easier to manage.

ChartDB's dashboard helps you control and monitor your schema visualizations effectively, all in one place.

✅ View and manage multiple database schemas in real-time.

✅ Customize schema layouts and export them in SQL or image format for easy sharing.

✅ Perform database migrations smoothly with AI-powered DDL script generation.

With ChartDB, you can simplify database design and migration tasks.

Check out ChartDB today and see how it transforms the way you work with databases!

Star ChartDB on GitHub ⭐️

9. Stack Auth - Managed user Authentication

The final tool on our list is Stack Auth, an open-source alternative to Auth0 and Clerk. If you're looking to integrate a self-hosted, customizable user authentication system, Stack Auth is an excellent choice, as its self-hosted version is completely free.

Stack Auth’s extensive support for features like single sign-on (SSO), OAuth, and multi-factor authentication (MFA).

It’s particularly suited for projects where you need to avoid vendor lock-in or reduce costs associated with third-party auth services.

The ability to self-host ensures that sensitive user data stays within your infrastructure, offering enhanced security for applications with strict compliance needs.

One interesting point to note is that Stack Auth provides a customizable, self-hosted authentication system, making it easy for developers to handle user access.

On the other hand, OPAL focuses on real-time policy management, ensuring that authorization rules remain aligned with changing application requirements.

Star StackAuth on GitHub ⭐️

That’s it! I've tried to cover a wide range of tools. if you know other awesome tools, write them in the comments!

Hope you found this article useful. If so, feel free to share it with your developer friends!

Also, Follow me For More Content like this: