A live version of my personal knowledge graph is available online.

How it works

Twice a day, a dedicated github workflow extracts:

• GitHub stars

• Twitter likes

• HackerNews upvotes

• Zotero records

Secrets

Twitter

To extract likes, we will need a Twitter API TOKEN, available on the Developer Portal after having creater an account.

TWITTER_TOKEN

Zotero

We will need a "group library" to index content from Zotero. The ZOTERO_API_KEY is available at https://www.zotero.org/settings/keys.

ZOTERO_API_KEY

The ZOTERO_LIBRARY_ID can be found by opening the group's page https://www.zotero.org/groups/groupname, and hovering over the group settings link. The ID is the integer after /groups/.

ZOTERO_LIBRARY_ID

Hackernews

We will need to create secrets for both Hackernews username and password.

HACKERNEWS_USERNAME

HACKERNEWS_PASSWORD

OpenAI

OpenAI API is used to call ChatGPT when pressing the button "ask" to re-rank documents based on our query. If we do not plan to use ChatGPT, we will need to set the secret OPENAI_API_KEY with an empty value. We can get our OpenAI key here.

OPENAI_API_KEY

The prompt to ChatGPT is stored in the api/api.py file.

Fly.io

We will need to install the flyctl client available here to set the FLY_API_TOKEN. The fly.io api token enables the github action to automatically push the updated state of the api. We can get the token using the command line:

flyctl auth signup

fly auth login

flyctl auth token

FLY_API_TOKEN

Sources

After finalizing the secrets, we can specify the Github and Twitter users whose liked content we wish to extract. To achieve this, we'll need to modify the sources.yml file located at the root of the repository. We'll be able to handpick the Github stars we want to index and set the Twitter ID and handle of the users whose content we want to include. To obtain the Twitter ID, we can use a tool like tweeterid.com."

github:
  - "raphaelsty"
  - "gbolmier"
  - "MaxHalford"
  - "AdilZouitine"

twitter:
  - [1262679654239961088, "raphaelsrty"]

Deployment

Fly.io

Once secrets and sources are set. We will deploy the API following the Fly.io documentation. You won't need any database. fly client should generate a fly.toml file that looks like the toml file below where app_name is the name of our api.

app = "app_name"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[experimental]
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 6
    soft_limit = 3
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

⚠️ After having created our API, we will need to update the urls called by the web app in the file docs/index.html. There are 3 urls to replace: https://knowledge.fly.dev per https://app_name.fly.dev where app_name is your API name.

Github Page

We will need to set the Github Page from the repository configurations (fork).

⚠️ After creating your github page, you will have to modify the origins field of the api/api.py file:

origins = [
    "https://raphaelsty.github.io", # Put your own github page name here.
]

Costs

⚠️ To avoid any financial incident, remember to define a hard_limit and a soft_limit which will bounder the number of instance Fly.io will deploy to answer to peak demands and therefore limit the costs. Those parameters are available in the fly.toml file.

[services.concurrency]
	hard_limit = 6
	soft_limit = 3
	type = "connections"

⚠️ Setting a 2GB memory VM with a single shared cpu on FLy.io will do the job for the app.

⚠️ Don't forget to define the limit amount you want to spend on OpenAI platform (10$ here).

Development

To run the API locally using Docker, we can export the OPENAI_API_KEY to our environment variables using:

export OPENAI_API_KEY="sk-..."

Then, we can run make launch at the root of the repository.

make launch

We can also deploy the API manually using:

fly deploy \
    --build-secret OPENAI_API_KEY=$OPENAI_API_KEY

Notes

My personal Knowledge Base is inspired and extract resources from the Knowledge Base of François-Paul Servant namely Semanlink.

License

GNU GENERAL PUBLIC LICENSE Knowledge Copyright (C) 2023 Raphaël Sourty