Upload packages already built locally how?

[akritkbehera]

Hi @pbuncic @ktf
This could be a easy question but if I build a package eg <Python> locally and it has the revision <local-1> assigned to it.
Then if I want to upload the same package to S3 would that mean again rebuilding the entire package?

[akritkbehera]

The recipe hasn't changed at all.

[pbuncic]

No re-build required if there is not change in the recipe, the package should be re-used from local cache and uploaded to S3 cache and CVMFS (if requested).

[akritkbehera]

It was built with revision prefix localok let me try why this isn't happening in my case. It rebuilds a new package without the prefix local and to upload it.

[pbuncic]

Which version of bits you are using?

[akritkbehera]

I am using the master from last week.

[pbuncic]

Can you provide reproducer script?

[pbuncic]

Perhaps there is a better way of achieving your goal of publishing build product to S3 and re-using them during next build (as of today, packages published by bits to CVMFS can also be re-used during subsequent builds, check the docs and recent issues).

Once bits finishes the build, it creates package manifest and stores it in work_dir (sw by default) under the same build_id. The entire set of build product can then be uploaded to S3 using a single command:

bits publish --from-manifest

This will do the bulk-uploads every locally-built package in a build manifest to the S3 store.

Prerequisites

boto3 on the build host (for a b3:// bucket): pip3 install --break-system-packages boto3, or the distro package (python3-boto3).

Credentials in ~/.bits/s3keys (chmod 600 — bits warns if it's group/other-readable). It accepts KEY=VALUE lines (optionally export-prefixed/quoted) or AWS-INI style. Recognised keys (with aliases): access key, secret key, and optionally session token, region, endpoint:

# ~/.bits/s3keys
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
# optional: S3_ENDPOINT_URL=https://s3.cern.ch, AWS_DEFAULT_REGION=...

A writable bucket.

In defaults-release.sh

system:
  remote_store:  "https://s3.cern.ch/swift/v1/<your bucket>"
  certify_group: "cms"
  manifests_remote: "https://gitlab.cern.ch/buncic/bits-manifests.git" # or clone bits-manifests repo so that you can have full control 

Commands

# 1. build locally (no store flags needed)
bits build xrootd 

# 2. upload the whole build to your bucket
bits publish --from-manifest --store b3://mybucket

--from-manifest with no path uses the newest manifest under sw/MANIFESTS; give a path to pick a specific one. Bare bits publish (no package, no --cvmfs-target) is the same thing — it defaults to --from-manifest latest and to s3 sore https://s3.cern.ch/lcgapp-bits-testing (or whatever you defined in defaults-relese.sh)

Only the packages built locally in that manifest are uploaded; deps that were reused from CVMFS are marked already-installed and skipped. An object already present in the bucket is skipped too. Files in store are stored with key corresponding to platform/hash (nothing to do with local build number that you mentioned).

Optional next step

Add --certify (or set system: certify_group: in your defaults) to also open a merge request in the manifests repo so the build's manifest gets signed and published for trusted reuse. That needs a GitLab PAT (--gitlab-token, or $BITS_CERTIFIER_TOKEN / ~/.bits/gitlab-token) — separate from the S3 keys.

That will trigger gitlab CI workflow that will verify that owner of token is listed as group admin, sign manifest with upload it to S3 bucket. Once file is listed in signed manifest it is eligible for re-use during next builds providing that architecture and hash match