Adds automated IPv6 database build and integration #231
Conversation
| # Combine lists | ||
| cat $GITHUB_WORKSPACE/input/${{matrix.list.name}}/ips/*.txt | grep -v '^#' | awk '{print $1}' | sed '/^$/d' > /tmp/manual-processed.txt | ||
| find $GITHUB_WORKSPACE/input/${{matrix.list.name}}/ips/ -type f -name "*.txt" ! -iname "*ipv6.txt" ! -iname "*IPv6.txt" -exec cat {} + | grep -v '^#' | awk '{print $1}' | sed '/^$/d' > /tmp/manual-processed.txt |
There was a problem hiding this comment.
In the future, we can organize IPv4 and IPv6 into different folders, for now I made a small fix.
| next if $line =~ /^#/ || $line =~ /^\s*$/; | ||
|
|
||
| # Basic IPv6 CIDR validation | ||
| if ($line =~ /^([0-9a-f:]+)\/(\d+)$/i) { |
There was a problem hiding this comment.
This also prevents any IPv4 address from being added to the list.
| awk -v asn="${asn:2}" '$3 == asn {print $1"-"$2}' /tmp/asndb-ipv6.tsv | while read -r range; do | ||
| start=$(echo "$range" | cut -d'-' -f1) | ||
| end=$(echo "$range" | cut -d'-' -f2) | ||
| python3 -c "import ipaddress; import sys; start=ipaddress.IPv6Address('$start'); end=ipaddress.IPv6Address('$end'); nets=list(ipaddress.summarize_address_range(start, end)); [print(str(net)) for net in nets]" >>/tmp/asn-processed-ipv6.txt |
There was a problem hiding this comment.
ipcalc does not support IPv6. Python is also better because we don't need to install any additional software or package, the default Python available on any normal operating system (💩 Windows) is enough.
| - name: Push Output of ipv6.txt to category folder | ||
| uses: X4BNet/copy_file_to_another_repo_action@main | ||
| env: | ||
| API_TOKEN_GITHUB: ${{ secrets.MY_GITHUB_TOKEN }} |
There was a problem hiding this comment.
You can also use build in "secrets.GITHUB_TOKEN"
There was a problem hiding this comment.
Aware its now possible to grant appropriate permissions to the default token. This wasnt possible when this was first written.
| matrix: | ||
| list: | ||
| - name: vpn | ||
| validate_low: 1 |
There was a problem hiding this comment.
This value can still be adjusted.
| destination_folder: '/output/${{matrix.list.name}}/' | ||
| user_email: '[email protected]' | ||
| user_name: 'listbuilder' | ||
| destination_branch: "main" |
There was a problem hiding this comment.
Perhaps a better solution would be to dynamically set the branch based on where the workflow is being executed? Currently, if I want to test something on a different branch, the data is still uploaded to main.
There was a problem hiding this comment.
Care needs to be taken to prevent both jobs from conflicting with eachother.
I would probably prefer a matrix build that does both IPv4 and IPv6 seperately and then a dependent job to update the repository.
This would also provide more CI re-use.
|
Nice work. I am fine with modernising and moving away from perl. I would like to re-write as a single JS application at some point. This would align with supporting some more listing methods (e.g rdns). Its been on my TODO list for a while. I don't think an IPv6 legacy path is required. At this point the IPv4 legacy path can likely be removed. Its been years. |
|
So, to summarize, what I need to do to merge the PR:
Is there anything else I need to do for this PR? |
2 participants
Hi,
Due to the fact that IPv6 is becoming more and more common in attacks, I have added the ability to generate IPv6 addresses from ASN. I've tried to maintain style similar to that of IPv4 scripts.
Everything is working, tested on other branch in my fork.
This should be probably enough to close #43
Of course, there are still few scripts to be added/improved (like fetching IPv6 from apple) but this can be done in another pull requests.
It's also possible to optimize the address retrieval and processing logic, we can add this to ToDo.
In my opinion, a better solution would be to create simple SH/Python scripts that would retrieve and process the database. Then, it could also be used in workflows. Currently, logic is somewhat duplicated because we have build-local scripts, perl (sic! it is 2025!) scripts to clean, ipcalc to calculate CIDR and a lot of other code in GitHub workflows.
IYKYK, next I will focus on adding support for IPv9 😆