Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feed sync #456

Draft
wants to merge 111 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
111 commits
Select commit Hold shift + click to select a range
7ec84df
Create gvm-feed-sync-docs
syspect-tech Jul 12, 2023
a03f3a0
Delete gvm-feed-sync-docs
syspect-tech Jul 12, 2023
4448f26
Create greenbone-feed-sync-docs.md
syspect-tech Jul 12, 2023
b112f6d
Update greenbone-feed-sync-docs.md
syspect-tech Jul 12, 2023
46d936a
Update greenbone-feed-sync-docs.md
syspect-tech Jul 12, 2023
9936598
Update greenbone-feed-sync-docs.md
syspect-tech Jul 13, 2023
a03379f
created feed-sync docs
syspect-tech Sep 14, 2023
693ad85
feed-sync changes
syspect-tech Oct 30, 2023
0688cc2
Removed the file and refernces to web interface documentation to keep…
syspect-tech Jan 10, 2024
9da852a
just temporarily stashing changes on hold
syspect-tech Jan 11, 2024
894013f
complete first-draft of Greenbone community feed sync documentation
syspect-sec Mar 6, 2024
9bfbac8
Added some changes to the Troubleshooting a connection section
syspect-sec Mar 7, 2024
0a5976c
Added warning label to all notices about Docker containers feed sync
syspect-sec Mar 19, 2024
bb7343a
Fix: Fix compose command for downloading gvmd data manually
bjoernricks Mar 15, 2024
9a29da6
Bump the python-packages group with 3 updates
dependabot[bot] Apr 1, 2024
2ff3c96
Bump idna from 3.6 to 3.7
dependabot[bot] Apr 12, 2024
598ae51
Add: openvas logs
nichtsfrei Apr 11, 2024
217ffdc
Add: openvasd Remove: notus
nichtsfrei Apr 11, 2024
881f3a4
Doc: remove notus-scanner and explain openvasd installation
nichtsfrei Apr 11, 2024
fbd9986
Update changelog.md
bjoernricks Apr 15, 2024
e0b4711
Fix: can not open or create log file
nichtsfrei Apr 17, 2024
ef15a28
OpenVasd Build Documation
Toys0125 Apr 17, 2024
046bcae
Bump the python-packages group with 2 updates
dependabot[bot] Apr 22, 2024
cd4ff0a
Change: Update docs for replacing notus-scanner with openvasd
bjoernricks Apr 19, 2024
de40eee
Add: openvasd container description
nichtsfrei Apr 19, 2024
86f60e0
Add: service description for openvasd
nichtsfrei Apr 19, 2024
a5432d6
Add: description of openvasd and it's goals
nichtsfrei Apr 19, 2024
844bee7
Add: openvasd configuration
nichtsfrei Apr 19, 2024
56f7bb3
Add: libcurl dependency to openvas-scanner
nichtsfrei Apr 24, 2024
77aafe5
Add: remove cargo from Debian and Centos
nichtsfrei Apr 24, 2024
21d6ee9
Add: scnnaerctl to openvasd, respect $INSTALL_DIR
nichtsfrei Apr 24, 2024
b4041ad
Stop and start openvasd via sytemctl too
bjoernricks Apr 24, 2024
646fb16
Fix: source-build: adds ospd-openvasd systemd dependency to openvasd.
nichtsfrei Apr 24, 2024
6cc0bc9
Fix captions for code block
bjoernricks Apr 24, 2024
7debfad
Bump the python-packages group with 3 updates
dependabot[bot] Apr 29, 2024
a9738ae
Update workflows.md
Nicolaj-H May 2, 2024
23327f7
Bump the python-packages group with 3 updates
dependabot[bot] May 6, 2024
46e3164
Small change to configuring gsad.service for remote access
syspect-sec May 6, 2024
a2b589d
Bump the python-packages group with 2 updates
dependabot[bot] May 13, 2024
f6aa8b0
Create gvm-feed-sync-docs
syspect-tech Jul 12, 2023
c9c382c
Delete gvm-feed-sync-docs
syspect-tech Jul 12, 2023
cfaa584
Create greenbone-feed-sync-docs.md
syspect-tech Jul 12, 2023
950d860
Update greenbone-feed-sync-docs.md
syspect-tech Jul 12, 2023
3f210d0
Update greenbone-feed-sync-docs.md
syspect-tech Jul 12, 2023
8e03c40
Update greenbone-feed-sync-docs.md
syspect-tech Jul 13, 2023
6baca0d
---
dependabot[bot] May 21, 2024
529a765
---
dependabot[bot] May 21, 2024
f7de60c
corrected the path for moving the openvasd build
syspect-sec May 21, 2024
a21dc39
Adjusted changelog.md
syspect-sec May 21, 2024
53e4449
Bump the python-packages group with 4 updates
dependabot[bot] May 27, 2024
e6d89ed
Fix: Typo in environment variable
n-thumann May 31, 2024
b5f3223
Fix: Typo in comment
n-thumann May 31, 2024
b7f10b7
Bump the python-packages group with 6 updates
dependabot[bot] Jun 3, 2024
2e6a1af
Bump the python-packages group with 3 updates
dependabot[bot] Jun 10, 2024
4a4c831
Bump urllib3 from 2.2.1 to 2.2.2
dependabot[bot] Jun 17, 2024
9c5f0a8
Bump the python-packages group across 1 directory with 2 updates
dependabot[bot] Jul 8, 2024
f948785
Bump exceptiongroup from 1.2.1 to 1.2.2 in the python-packages group
dependabot[bot] Jul 15, 2024
9d92ac4
Fix color of "view this page" icon in dark mode (#489)
k-schlosser Jul 19, 2024
97071b1
Change color of visited links
k-schlosser Jul 19, 2024
f3cbf90
Remove MQTT broker from architecture image
k-schlosser Jul 25, 2024
d3b38d8
Update version of ospd-openvas (#494)
k-schlosser Jul 26, 2024
11cf575
Remove arrow between Notus scanner and target systems in architecture…
k-schlosser Jul 26, 2024
d10212d
Bump the python-packages group across 1 directory with 10 updates
dependabot[bot] Jul 29, 2024
80f745f
Update versions of source-build components
k-schlosser Aug 1, 2024
4f96cd7
Updated line 62 and 202 with ne registry
gabrielschowe Aug 5, 2024
eaa415e
Bump uvicorn from 0.30.3 to 0.30.5 in the python-packages group
dependabot[bot] Aug 5, 2024
35b5954
Revert "Updated line 62 and 202 with ne registry"
y0urself Aug 6, 2024
4e757fc
Bump the python-packages group with 5 updates
dependabot[bot] Aug 12, 2024
2397d87
Update src/feed-sync.md
syspect-sec Aug 16, 2024
aec90f7
Update src/feed-sync.md
syspect-sec Aug 16, 2024
c25d8d4
Update src/feed-sync.md
syspect-sec Aug 16, 2024
ef915da
Update src/feed-sync.md
syspect-sec Aug 16, 2024
55347bc
Update src/feed-sync.md
syspect-sec Aug 16, 2024
558ffc4
Update src/feed-sync.md
syspect-sec Aug 16, 2024
14e15a2
Update src/feed-sync.md
syspect-sec Aug 16, 2024
e0f351c
Update src/feed-sync.md
syspect-sec Aug 16, 2024
aac3c42
Update src/feed-sync.md
syspect-sec Aug 16, 2024
fea2ceb
Update src/feed-sync.md
syspect-sec Aug 16, 2024
8670810
Update src/feed-sync.md
syspect-sec Aug 16, 2024
4cddf2f
Update src/feed-sync.md
syspect-sec Aug 16, 2024
59cd4cb
Update src/feed-sync.md
syspect-sec Aug 16, 2024
c8b1bb4
Update src/feed-sync.md
syspect-sec Aug 16, 2024
8844977
Update src/feed-sync.md
syspect-sec Aug 16, 2024
c4e12a1
Update src/feed-sync.md
syspect-sec Aug 16, 2024
8aa92ae
Update src/feed-sync.md
syspect-sec Aug 16, 2024
deb3378
Update src/feed-sync.md
syspect-sec Aug 16, 2024
f5216d8
Apply suggestions from code review
syspect-sec Aug 16, 2024
654e742
Create gvm-feed-sync-docs
syspect-tech Jul 12, 2023
0cc42ab
Delete gvm-feed-sync-docs
syspect-tech Jul 12, 2023
e1399ff
Create greenbone-feed-sync-docs.md
syspect-tech Jul 12, 2023
9449622
Update greenbone-feed-sync-docs.md
syspect-tech Jul 12, 2023
2bdd6bb
Update greenbone-feed-sync-docs.md
syspect-tech Jul 12, 2023
973d54d
Update greenbone-feed-sync-docs.md
syspect-tech Jul 13, 2023
e3b422b
created feed-sync docs
syspect-tech Sep 14, 2023
f888b01
feed-sync changes
syspect-tech Oct 30, 2023
87138bd
Removed the file and refernces to web interface documentation to keep…
syspect-tech Jan 10, 2024
0f4122a
just temporarily stashing changes on hold
syspect-tech Jan 11, 2024
a62b20a
complete first-draft of Greenbone community feed sync documentation
syspect-sec Mar 6, 2024
526a072
Added some changes to the Troubleshooting a connection section
syspect-sec Mar 7, 2024
b6fdd97
Added warning label to all notices about Docker containers feed sync
syspect-sec Mar 19, 2024
e993ec2
Merge branch 'main' of https://github.com/greenbone/docs
syspect-sec Aug 19, 2024
a0fd77a
Merge branch 'main' into feed-sync
syspect-sec Aug 19, 2024
9f8335d
Fixed introduction paragraph for clarify and malformed tags for a not…
syspect-sec Aug 19, 2024
7144b1e
changelog merge conflict fixed
syspect-sec Aug 19, 2024
238b055
commit conflics before merge with upstream main changes
syspect-sec Aug 19, 2024
f5238c2
fixed problems with the src/feed-sync.md file. Some fragments of the…
syspect-sec Aug 19, 2024
c55e4a4
Changed the context of the introduction paragraph to refer to a singl…
syspect-sec Aug 19, 2024
c6ea3b7
Update greenbone-feed-sync-docs.md
k-schlosser Aug 26, 2024
849c680
Update greenbone-feed-sync-docs.md
k-schlosser Aug 26, 2024
53fcdf8
Update greenbone-feed-sync-docs.md
k-schlosser Aug 26, 2024
04e1309
Update greenbone-feed-sync-docs.md
k-schlosser Aug 26, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Apply suggestions from code review
Batch commit for changes

Co-authored-by: Kristin Schlosser <57938820+k-schlosser@users.noreply.github.com>
  • Loading branch information
syspect-sec and k-schlosser authored Aug 16, 2024
commit f5216d8d906a3d3174524670518f96f9d6982f88
58 changes: 35 additions & 23 deletions src/feed-sync.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,39 +52,45 @@ Python 3.9 and later is supported. Also note that syncing feeds requires the `rs

#### Installation Using pipx
You can install the latest stable release of **greenbone-feed-sync** from the Python Package Index (pypi) using [pipx](https://pypa.github.io/pipx/).
syspect-sec marked this conversation as resolved.
Show resolved Hide resolved

```{code-block}
:caption: Installing the package via pipx
python3 -m pipx install greenbone-feed-sync
```

#### Install using pip
NOTE: The pip install command no longer works out-of-the-box in newer distributions like Ubuntu 23.04 because of [PEP 668](https://peps.python.org/pep-0668). Please use the [installation via pipx](https://github.com/greenbone/greenbone-feed-sync/tree/main#install-using-pipx) instead.
#### Installation Using pip
:::{note}
The pip install command no longer works out-of-the-box in newer distributions like Ubuntu 23.04 because of [PEP 668](https://peps.python.org/pep-0668). Please use the [installation via pipx](https://github.com/greenbone/greenbone-feed-sync/tree/main#install-using-pipx) instead.

You can install the latest stable release of **greenbone-feed-sync** from the Python Package Index (pypi) using [pip](https://pip.pypa.io/).
syspect-sec marked this conversation as resolved.
Show resolved Hide resolved

```{code-block}
:caption: Installing the package via python-pip
python3 -m pip install greenbone-feed-sync
```

## Updating the Greenbone Feeds
### Updating Feeds For Greenbone Docker Containers
### Updating Feeds for Greenbone Docker Containers
```{warning} Please note that different methods are required for completing a feed sync when using the official **Greenbone Docker containers**, which can be [found here](https://greenbone.github.io/docs/latest/22.4/container/workflows.html#performing-a-feed-synchronization).
```

### Determining The Script Version In Use
If you are unsure of which version of the feed sync script is active, you should check before you execute the `greenbone-feed-sync` command.One way to check is by using the `--version` argument. The legacy bash scripts will output the GVMD version, while the new Python scripts will output the version of the `greenbone-feed-sync` package itself. Otherwise, the script header can help distinguish which file is in use with certainty.
### Determining the Script Version in Use
If you are unsure of which version of the feed sync script is active, you should check this before executing the `greenbone-feed-sync` command. One way to check is by using the `--version` argument. The legacy bash scripts will output the GVMD version, while the new Python scripts will output the version of the `greenbone-feed-sync` package itself. Otherwise, the script header can help distinguish which file is in use with certainty.

To check the file header, use the `which` command to determine the path of the command that is executed.

For example:
syspect-sec marked this conversation as resolved.
Show resolved Hide resolved

```{code-block}
:caption: Check the path of the active feed sync script
:caption: Checking the path of the active feed sync script
which greenbone-feed-sync
/usr/local/bin/greenbone-feed-sync
```
Once you determine the path of the executed command you can identify whether it is the new Python-based feed sync command or the legacy bash version. To do that, use the `head` command to print the first line of the script. This will indicate the interpreter used at run-time.

Once you determined the path of the executed command, you can identify whether it is the new Python-based feed sync command or the legacy bash version. To do that, use the `head` command to print the first line of the script. This will indicate the interpreter used at run-time.

For example:
syspect-sec marked this conversation as resolved.
Show resolved Hide resolved

```{code-block}
:caption: If the new Python-based feed update script is active you will see
head -1 /usr/local/bin/greenbone-feed-sync
Expand All @@ -97,17 +103,19 @@ head -1 /usr/local/sbin/greenbone-feed-sync
```

### Updating Feeds With New Python Script
By default, when no arguments are passed, the new Python `greenbone-feed-sync` will updated all feed types. For standard installations that follow the [official Greenbone source code install instructions](https://greenbone.github.io/docs/latest/22.4/source-build/index.html), or the [Kali Linux native package installation](https://greenbone.github.io/docs/latest/22.4/kali/index.html), the new Python script should be run with `root`/`sudo` permissions, as it will change its execution context appropriately to the `gvm` or `_gvm` user.
By default, when no arguments are passed, the new Python `greenbone-feed-sync` will update all feed types. For standard installations that follow the [official Greenbone source-code install instructions](https://greenbone.github.io/docs/latest/22.4/source-build/index.html), or the [Kali Linux native package installation](https://greenbone.github.io/docs/latest/22.4/kali/index.html), the new Python script should be run with `root`/`sudo` permissions, as it will change its execution context appropriately to the `gvm` or `_gvm` user.

However, for non-standard installations that use a different username and group to execute the Greenbone components, the configured username and group context must be changed via the `greenbone-feed-sync.toml` configuration file. A basic example of how to configure a non-standard user via the `.toml` file can be [found here](https://github.com/greenbone/greenbone-feed-sync/blob/main/README.md#usage-on-kali-linux).

```{code-block}
:caption: Update all feeds with single command
:caption: Updating all feeds with single command
sudo greenbone-feed-sync
```

To update only one feed at a time, the new Python-based command can be used similar to the previous `/usr/sbin/greenbone-feed-sync` bash script command by specifying a feed type.
syspect-sec marked this conversation as resolved.
Show resolved Hide resolved

```{code-block}
:caption: Include the feed type as an argument to update only a specific feed type
:caption: Including the feed type as an argument to update only a specific feed type
sudo greenbone-feed-sync --type <feed-type>
```

Expand All @@ -119,9 +127,9 @@ The available feed types are:
* `gvmd-data`: Synchronizes GVM data feed which includes port lists, scan configs and report formats
* `scap`: Synchronizes SCAP data feed
* `cert`: Synchronizes CERT data feed
* `notus`: Fetch any newly available NOTUS vulnerability tests
* `nasl`: Fetch any newly available NASL vulnerability tests
* `report-format` or `report-formats`: Fetches any newly available scan report formats
* `notus`: Fetches any newly available Notus vulnerability tests
* `nasl`: Fetches any newly available NASL vulnerability tests
* `report-format` or `report-formats`: Fetches any newly available report formats
* `scan-config` or `scan-configs`: Fetches any newly available scan configurations
* `port-list` or `port-lists`: Fetches any newly available port lists

Expand All @@ -137,7 +145,7 @@ A complete list of settings are [available here](https://github.com/greenbone/gr
The most basic arguments are:

* `--help`: Displays list of all available command line arguments
* `--verbose`: Sets verbose output. `-vvv` provides maximum verbosity.
* `--verbose`: Sets verbose output (`-vvv` provides maximum verbosity)
* `--version`: Displays the version of the script
* `--identify`: Displays information about the script, including script ID, name, version, feed name, whether it is restricted or not
* `--describe`: Displays a description of the script and the feed it synchronizes
Expand All @@ -146,22 +154,22 @@ The most basic arguments are:
* `--feedcurrent`: Checks if the feed is already up to date

### Updating Feeds With Legacy Bash Scripts (Not Recommended)
If you absolutely cannot update to the new Python-based feed sync scripts because you are using a version of Greenbone prior to 22.4, or another reason, it's important to note that the legacy bash scripts can be run as the `sudo`/`root` user or the `gvm` user standard installations (`_gvm` on legacy versions of Kali). However, for non-standard installations that use a different username for the Greenbone components, the feed sync script should be run from the appropriate user context. Also, to update the feed sync using the legacy `greenbone-feed-sync` command, the `--type` parameter is **required**.
If you absolutely cannot update to the new Python-based feed sync scripts because you are using a version of Greenbone prior to 22.4, or another reason, it is important to note that the legacy bash scripts can be run as the `sudo`/`root` user or the `gvm` user standard installations (`_gvm` on legacy versions of Kali). However, for non-standard installations that use a different username for the Greenbone components, the feed sync script should be run from the appropriate user context. Also, to update the feed sync using the legacy `greenbone-feed-sync` command, the `--type` parameter is **required**.

For example:

```{code-block}
:caption: Update feeds with legacy bash script
:caption: Updating feeds with legacy bash script
sudo -u gvm greenbone-feed-sync --type <feed-type>
```

Using one of the following values to replace **\<feed-type>**:
Use one of the following values to replace **\<feed-type>**:

* `CERT`: Synchronizes CERT data feed
* `SCAP`: Synchronizes SCAP data feed
* `GVMD_DATA`: Synchronizes gvmd data feed

### Updating Feeds For Kali Linux Native Installation
### Updating Feeds for Kali Linux Native Installation
Some older versions of Kali Linux include the `gvm-feed-update` command for feed synchronization. However, the wrapper script has been depreciated in newer versions of Kali and Greenbone's official `greenbone-feed-sync` command must be used instead.

To update all feeds for versions of Kali Linux that include the `gvm-feed-update` script issue the following command can be used:
Expand All @@ -173,25 +181,28 @@ To update all feeds for versions of Kali Linux that include the `gvm-feed-update

Please note that the `gvm-feed-update` command mentioned above is not maintained by Greenbone. It is maintained by the Kali/Debian package managers. Under the hood, the `gvm-feed-update` is a bash script that essentially executes the standard `greenbone-feed-sync --type <feed-type>` scripts provided by Greenbone. We recommend avoiding the use of `gvm-feed-update` for your feed sync operations.

### Configuring The Feed Sync User For Non-Standard Installations
### Configuring the Feed Sync User for Non-Standard Installations
The Python-based feed sync scripts will automatically change context to the `gvm` user in order to process the feeds. Since the Kali native installation uses the `_gvm` user, it comes preconfigured with a `.toml` file which specifies the appropriate user context. The `greenbone-feed-sync` script will look for this configuration file at the locations `~/.config/greenbone-feed-sync.toml` and `/etc/gvm/greenbone-feed-sync.toml`. If you have configured a Greenbone source code installation with a non-standard username and group such as `_gvm` you must add the user and group context to the `/etc/gvm/greenbone-feed-sync.toml` file, configure the appropriate environment variable, or supply the user context via the `--user` and `--group` command line arguments.

### Verifying Successful Feed Sync Update
After the sync operation completes, you can verify the successful update of the feed data by visiting the `/feedstatus` page which can be found under the **Administration** item in the top menu bar of Greenbone's web-interface. If a feed has successfully updated you will see **Current** beside the particular feed and the **Version** column will indicate the date/time of the feed's release.
After the sync operation completes, you can verify the successful update of the feed data by visiting the *Feed Status* page which can be found under *Administration* in the menu bar of GSA. If a feed was successfully updated you will see **Current** beside the particular feed and the **Version** column will indicate the date/time of the feed's release.

## Automating Feed Sync Updates
The Greenbone feeds are not automatically synced for the Greenbone Community source-code installations, Greenbone Community Docker containers, or Greenbone native Kali Linux installation. The suggested method for automating the Greenbone feed sync is to create a `crontab` file to schedule the feed sync command execution. For standard source-code installations, the `crontab` should be run under the `sudo` or `root` user context.

```{warning} Please note that different methods are required for completing a feed sync when using the official **Greenbone Docker containers**, which can be [found here](https://greenbone.github.io/docs/latest/22.4/container/workflows.html#performing-a-feed-synchronization). These commands can also be scheduled via **crontab**.
```

## Troubleshooting A Connection To The Greenbone Feeds
To trouble shoot a connection to the Greenbone feeds, you can issue the following command from a terminal.
## Troubleshooting a Connection to the Greenbone Feeds
To troubleshoot a connection to the Greenbone feeds, you can issue the following command from a terminal.

```{code-block}
:caption: Verify an rsync connection to the Greenbone feeds
:caption: Verifying an rsync connection to the Greenbone feeds
rsync rsync://feed.community.greenbone.net/community
```

You should see output such as the following:
syspect-sec marked this conversation as resolved.
Show resolved Hide resolved

```{code-block}
:caption: Expected output when verifying an rsync connection to the Greenbone feeds
Greenbone community feed server - http://feed.community.greenbone.net/
Expand All @@ -213,6 +224,7 @@ cert-data Greenbone community CERT data feed, see https://community.greenbone.n
community-legacy Greenbone community legacy feed, see https://community.greenbone.net/
community Greenbone community feed, see https://community.greenbone.net/
```

If you are unable to connect to the Greenbone feeds using the `rsync` command, we suggest you troubleshoot your network connection to identify any firewalls, content proxies, or network configurations that could be blocking the connection. Here are some suggested methods:
syspect-sec marked this conversation as resolved.
Show resolved Hide resolved

### Using nping
Expand Down