You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Pycolytics is written in python, based on [SQLite](https://github.com/sqlite/sqlite) and [FastAPI](https://github.com/fastapi/fastapi), and was inspired by [Attolytics](https://github.com/ttencate/attolytics/).
25
-
26
-
When I was looking at Attolytics, I was too lazy to set up a rust compile environment and install postgresql for something so simple, so I spent two days writing Pycolytics instead. To help you avoid my mistake, I made it so you can just clone it and move on with your life.
17
+
- Launches out of the box:
27
18
28
-
True to its name, Pycolytics is probably 10<sup>6</sup> times slower than Attolytics, but who cares if it still serves my entire userbase from a rasberry-pi. It does asyncio and fancy multi-worker stuff to try and compensate.
19
+
```sh
20
+
fastapi dev
21
+
```
29
22
30
-
Open an issue if you wish to contribute, or buy me a coffee if you find my work useful.
31
-
32
-
<a href='https://ko-fi.com/E1E712JJXK' target='_blank'><img height='36' style='border:0px;height:36px;' src='https://storage.ko-fi.com/cdn/kofi3.png?v=3' border='0' alt='Buy Me a Coffee at ko-fi.com' /></a>
23
+
- After launch, API docs are available at: <http://127.0.0.1:8000/docs>
33
24
25
+
Pycolytics is written in python, based on
26
+
[SQLite](https://github.com/sqlite/sqlite) and
27
+
[FastAPI](https://github.com/fastapi/fastapi), and was inspired by
- I wrote a plugin for Godot: just install it and call a single function to log an event! You can find it in the [asset library](https://godotengine.org/asset-library/asset/3292), or on [github](https://github.com/KerekesDavid/pycolytics-godot).
37
31
38
-
- If you have written clients for anything else, I would be more than happy to feature them here!
32
+
- I wrote a plugin for Godot: just install it and call a single function to log
33
+
an event! You can find it in the
34
+
[asset library](https://godotengine.org/asset-library/asset/3292), or on
- If you have written clients for anything else, I would be more than happy to
38
+
feature them here!
39
39
40
40
## Configuration
41
+
41
42
Edit the .env file, or specify these parameters as environment variables:
42
-
```
43
+
44
+
```sh
43
45
# Name of the database file to write into.
44
46
SQLITE_FILE_PATH="databases/database.db"
45
47
46
-
# A list of secret keys. The server won't accept events that do not contain one of these keys in the request body.
48
+
# A list of secret keys. The server won't accept events
49
+
# that do not contain one of these keys in the request body.
47
50
API_KEYS=["I-am-an-unsecure-dev-key-REPLACE_ME"]
48
51
49
52
# Requests from the same IP above this rate will be rejected.
@@ -52,9 +55,11 @@ RATE_LIMIT="60/minute"
52
55
```
53
56
54
57
## API
55
-
The server will listen to POST requests at `http://ip:port/v1.0/event`, and will expect a request body in the following format:
56
58
57
-
```
59
+
The server will listen to POST requests at `http://ip:port/v1.0/event`,
60
+
and will expect a request body in the following format:
61
+
62
+
```json
58
63
{
59
64
"event_type": "string",
60
65
"application": "string",
@@ -65,23 +70,25 @@ The server will listen to POST requests at `http://ip:port/v1.0/event`, and will
65
70
"value": {
66
71
"event_description": "Life, the universe and everything.",
67
72
"event_data": 42
68
-
},
73
+
},
69
74
"api_key": "I-am-an-unsecure-dev-key-REPLACE_ME"
70
75
}
71
76
```
72
77
73
-
There is also a more performant batch interface at `http://ip:port/v1.0/events`, expecting a list of events:
74
-
```
78
+
There is also a more performant batch interface at
79
+
`http://ip:port/v1.0/events`, expecting a list of events:
80
+
81
+
```json
75
82
[
76
-
{"event_type": ...},
83
+
{"event_type": ...},
77
84
{"event_type": ...},
78
85
...
79
86
]
80
87
```
81
88
82
89
An example curl call for logging an event:
83
90
84
-
```
91
+
```sh
85
92
curl -X 'POST' \
86
93
'http://127.0.0.1:8000/v1.0/event' \
87
94
-H 'accept: */*' \
@@ -106,9 +113,11 @@ The `value` field can contain an arbitrary JSON with event details.
106
113
The POST request will return `204: No Content` on successful inserts.
107
114
108
115
## Database
116
+
109
117
The database will contain an `event` table with all logged events.
110
118
The columns are:
111
-
```
119
+
120
+
```sql
112
121
event_type VARCHARNOT NULL
113
122
platform VARCHARNOT NULL
114
123
version VARCHARNOT NULL
@@ -119,82 +128,116 @@ id INTEGER NOT NULL, PRIMARY KEY
119
128
time DATETIME NOT NULL
120
129
```
121
130
122
-
It can be opened using any sqlite database browser, or in python using the built in sqlite package.
131
+
It can be opened using any sqlite database browser,
132
+
or in python using the built in sqlite package.
123
133
124
-
My personal choice for performing data analytics is a [jupyter notebook](https://jupyter.org/) using [pandas](https://pandas.pydata.org/). They have a wonerful cheat sheet [here](https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf).
134
+
My personal choice for performing data analytics is a
135
+
[jupyter notebook](https://jupyter.org/) using
136
+
[pandas](https://pandas.pydata.org/). They have a wonderful cheat sheet
Setting up a permanent server as a service is also quite simple.
128
142
129
-
The method I share here has some minimal extra complications, but it ensures some level of separation from other parts of the system using systemd's `DynamicUser` parameter. It might come handy in case there is a vulnerability in FastAPI.
143
+
The method I share here has some minimal extra complications, but it ensures
144
+
some level of separation from other parts of the system using systemd's
145
+
`DynamicUser` parameter. It might come handy in case there is a vulnerability
146
+
in FastAPI.
130
147
131
-
(Contributions to this section are very welcome, I'm barely a fledgeling server admin.)
148
+
(Contributions to this section are very welcome, I'm barely a fledgelig server admin.)
132
149
133
150
- Install pycolytics, and set up a virtualenv.
134
151
A usual place for this would be `/srv/pycolytics` for example.
135
152
136
153
- Create a systemd service file: `/etc/systemd/system/pycolytics.service`
137
154
138
-
```
139
-
[Unit]
140
-
Description=Uvicorn instance serving Pycolytics
141
-
After=network.target
155
+
```INI
156
+
[Unit]
157
+
Description=Uvicorn instance serving Pycolytics
158
+
After=network.target
142
159
143
-
[Service]
144
-
Type=simple
145
-
DynamicUser=yes
146
-
User=pycolytics
160
+
[Service]
161
+
Type=simple
162
+
DynamicUser=yes
163
+
User=pycolytics
147
164
148
-
WorkingDirectory=/srv/pycolytics
149
-
StateDirectory=pycolytics/databases
165
+
WorkingDirectory=/srv/pycolytics
166
+
StateDirectory=pycolytics/databases
150
167
151
-
ExecStart=/srv/pycolytics/.venv/bin/uvicorn \
152
-
--workers=4 \
153
-
--host=0.0.0.0 \
154
-
--port=8080 \
155
-
app.main:app
156
-
ExecReload=/bin/kill -HUP ${MAINPID}
157
-
RestartSec=15
158
-
Restart=always
168
+
ExecStart=/srv/pycolytics/.venv/bin/uvicorn \
169
+
--workers=4 \
170
+
--host=0.0.0.0 \
171
+
--port=8080 \
172
+
app.main:app
173
+
ExecReload=/bin/kill -HUP ${MAINPID}
174
+
RestartSec=15
175
+
Restart=always
159
176
160
-
[Install]
161
-
WantedBy=multi-user.target
177
+
[Install]
178
+
WantedBy=multi-user.target
179
+
180
+
```
162
181
163
-
```
164
182
- Generate an API key:
165
-
```
166
-
openssl rand -base64 24
167
-
```
168
-
This will stop random people from logging events in your database, it will not stop a script kiddie who can decompile the key from your app, or pluck it from network traffic. I'd suggest creating a new one for every version of your application, and retiring old ones after a while.
169
183
170
-
- Setup the .env file:
171
-
- Replace `API_KEYS=["I-am-an-unsecure-dev-key-REPLACE_ME"]` with the newly generated key.
172
-
- Set the database path to the systemd state directory: `SQLITE_FILE_NAME="/var/lib/pycolytics/databases/database.db"`
184
+
```sh
185
+
openssl rand -base64 24
186
+
```
187
+
188
+
This will stop random people from logging events in your database, it will
189
+
not stop a script kiddie who can decompile the key from your app, or pluck it
190
+
from network traffic. I'd suggest creating a new one for every version of
191
+
your application, and retiring old ones after a while.
192
+
193
+
- Set up the .env file:
194
+
195
+
- Replace `API_KEYS=["I-am-an-unsecure-dev-key-REPLACE_ME"]` with the
196
+
newly generated key.
197
+
- Set the database path to the systemd state directory:
- In case you need to fix configurations and restart the service use:
191
217
192
-
```sudo systemctl daemon-reload
193
-
sudo systemctl restart pycolytics
194
-
```
218
+
```sudo systemctl daemon-reload
219
+
sudo systemctl restart pycolytics
220
+
```
195
221
196
-
Most online guides also recommend setting up fastapi behind an nginx reverse proxy, in case somebody tries to DDOS your server. I've never been successful enough for this to happen, so I'll leave it to you to figure out the details.
222
+
Most online guides also recommend setting up fastapi behind an nginx reverse
223
+
proxy, in case somebody tries to DDOS your server. I've never been successful
224
+
enough for this to happen, so I'll leave it to you to figure out the details.
225
+
226
+
## How We Got Here
227
+
228
+
When I was looking at Attolytics, I was too lazy to set up a rust compile
229
+
environment and install postgresql for something so simple, so I spent two days
230
+
writing Pycolytics instead. To help you avoid my mistake, I made it so you can
231
+
just clone it and move on with your life.
232
+
233
+
True to its name, Pycolytics is probably 10⁶ times slower than Attolytics, but
234
+
who cares if it still serves my entire userbase from a raspberry-pi. It does
235
+
asyncio and fancy multi-worker stuff to try and compensate.
236
+
237
+
Open an issue if you wish to contribute, or buy me a coffee if you find my work useful.
238
+
239
+
<ahref='https://ko-fi.com/E1E712JJXK'target='_blank'><imgheight='36'style='border:0px;height:36px;'src='https://storage.ko-fi.com/cdn/kofi3.png?v=3'border='0'alt='Buy Me a Coffee at ko-fi.com' /></a>
197
240
198
241
## Planned Features
199
-
- HTTPS communication for you security nerds out there
200
242
243
+
- HTTPS communication for you security nerds out there
0 commit comments