feat(iaas): add stackit_networks data source

[thorbenbelow]

Description

This PR adds the stackit_networks data source that allows listing Networks in a project, optionally filtered by name and/or labels.

The filtering options are modeled after the existing patterns in stackit_service_accounts. The output is returned as a simplified flat list containing only the network name and network ID to avoid code duplication, similar to plural data sources in other providers (e. g. aws_subnets).

solves #1600

Checklist

• Issue was linked above
• Code format was applied: make fmt
• Examples were added / adjusted (see examples/ directory)
• Docs are up-to-date: make generate-docs (will be checked by CI)
• Unit tests got implemented or updated
• Acceptance tests got implemented or updated (see e.g. here)
• Unit tests are passing: make test (will be checked by CI)
• No linter issues: make lint (will be checked by CI)

[github-actions]

This PR was marked as stale after 7 days of inactivity and will be closed after another 7 days of further inactivity. If this PR should be kept open, just add a comment, remove the stale label or push new commits to it.

[github-actions]

This PR was marked as stale after 7 days of inactivity and will be closed after another 7 days of further inactivity. If this PR should be kept open, just add a comment, remove the stale label or push new commits to it.

[github-actions]

This PR was marked as stale after 7 days of inactivity and will be closed after another 7 days of further inactivity. If this PR should be kept open, just add a comment, remove the stale label or push new commits to it.

[rubenhoenle]

Hi @thorbenbelow ,

first of all thanks for the contribution and sorry that it took some time to get some eyes on it from a maintainer.

In general I appreciate the implementation of a stackit_networks datasource and would be more than happy to continue with this PR. Nevertheless there's one catch and that's the filtering option.

We from the STACKIT Developer Tools team did some research and planning how we want to continue with the filtering options in such list datasources like e.g. your new stackit_networks datasource but also others like stackit_ske_kubernetes_versions, stackit_routing_tables, ...

We decided that solving this problem of missing filter options in the STACKIT Terraform provider on the client side isn't the way to go. Even more, we consider it as technical debt which would be introduced here.

The reasons are the following (just a short breakdown, not a complete list):

• For client side filtering you need to load all entries using the list endpoint from the STACKIT APIs before applying the filter values
  • If you have e.g. many resources in your STACKIT project the list endpoint will force you to use pagination at some point to handle the large amount of resources properly.
    • Now you have to handle the pagination client side (which is needed anyways, but it will slow down the whole Terraform provider) but you're also doing way more API calls than needed.
    • The potential large amount of API calls caused by the pagination will possibly have the effect that you'll run into rate limits. Then the STACKIT Terraform provider needs to slow down with the amount of API requests and wait for some time. This will slow down STACKIT Terraform provider runs even more.
• Client side filtering in the Terraform provider doesn't scale
  • On the one hand there's the maintenance of the filters which has to be done in the STACKIT Terraform provider codebase for all STACKIT products
  • On the other hand implementing filters on client side in the Terraform provider won't make them available e.g. in the STACKIT CLI.
• Client-side filtering is very limited: There are use cases where you want to combine filters using logical operators, let's say e.g. for listing database flavors. Users might want to filter for flavors which have at least 4 CPU cores and exactly 8 GB of RAM. This is hard to implement on client side - technically possible of course, but a pain to implement, test and maintain. Solving this within the corresponding STACKIT API is easy since one could just solve this using e.g. a SQL query on server side.

The solution we want to go with is server-side filtering using the STACKIT APIs. There's some standardized way of implementing filter options in STACKIT APIs written down in an AIP (API improvment proposal).

In Terraform it would look like this in the future (in the example we're listing IaaS machine types):

data "stackit_machine_types" "single_core" {
  project_id = var.project_id
  # filters for machine types with 1 vCPU and 1024 MB of RAM
  # for first time users who want to use simple filters based on vCPU and RAM
  filter = {
    vcpu = 1
    ram  = 1024
  }
}

data "stackit_machine_types" "dual_core" {
  project_id = var.project_id
  # filters for machine types with at least 2 vCPU and at least 1024 MB of RAM
  # for power users who want to use more specific filters
  filter = {
    query = "vcpus == 2 AND ram >= 1024"
  }
}

As soon as a STACKIT service implements the filter options based on the AIP we will integrate them in our STACKIT Terraform provider.

So I would propose you'll remove the filter options for now from the datasource implementation so we can go ahead and merge your new stackit_networks datasource. In the meantime you can rely on HCL to filter your networks.

Best regards

Ruben from STACKIT Developer Tools team

[rubenhoenle]

@thorbenbelow if you have any questions just post them in this PR and I'll comment them.

[thorbenbelow]

Hi @rubenhoenle,
thanks for the detailed explanation and for taking the time to share the reasoning behind the decision.

Not implementing the filters client side definitely makes sense.
Also, the filter standardization approach you outlined looks pretty nice. Looking forward to that!

Regarding the filter options:
The datasource currently includes a label selector, which is already evaluated server side. Would you be okay with keeping that part?

data "stackit_networks" "example" {
  project_id = var.project_id

  filter = {
    labels = {
      foo = "bar"
    }
  }
}