Skip to content

feat(auto-tuing): add a new auto-tuning system - #879

Open
mingdaw689 wants to merge 7 commits into
InfiniTensor:masterfrom
mingdaw689:feat/auto-tuning
Open

feat(auto-tuing): add a new auto-tuning system#879
mingdaw689 wants to merge 7 commits into
InfiniTensor:masterfrom
mingdaw689:feat/auto-tuning

Conversation

@mingdaw689

@mingdaw689 mingdaw689 commented Aug 4, 2026

Copy link
Copy Markdown

Summary

This PR implements runtime auto-tuning for InfiniOps operators, enabling automatic selection of the fastest implementation based on live benchmarking.

  • src/tuning.h / src/tuning.cc — Core tuning infrastructure: TuningSignature extracts shape/dtype/scalars from operator arguments via C++17 fold expressions; TuningManager singleton handles lookup, recording, and persistence to tuning.json with mutex-guarded concurrent access
  • src/tuning_utils.h — Helper utilities: operator name extraction from template types via __PRETTY_FUNCTION__, environment variable parsing for warmup/repeat counts, and first-tensor device type inference
  • src/operator.h — Added ResolveConfigOnline<Key>() called at Operator::Call entry: queries cache, benchmarks candidates via BenchmarkImplementation<Key>() on miss, records winner; added detail::SyncDevice() for accurate GPU timing and detail::ExtractOperatorName<Key>() for runtime operator identification
  • src/config.h — Added auto_select_ member and accessor; set_implementation_index() now sets auto_select_ = false to distinguish user-specified indices from auto-selection
  • src/CMakeLists.txt — Added tuning.cc to build (unconditional compilation)
  • scripts/generate_wrappers.py — Modified Python binding generation: only calls set_implementation_index() when user explicitly passes the parameter, preserving auto_select_=true for tuning; added TuningManager::Instance().LoadTuningCache() call in PYBIND11_MODULE to load existing records at import time
  • First call with a new input shape benchmarks all active implementations (warmup + repeat), selects the fastest, and writes to tuning.json
  • Subsequent calls with the same shape read from cache (zero overhead)
  • Single-implementation operators record their index without benchmarking
  • Explicit implementation_index parameter bypasses auto-tuning entirely

Motivation

In the operator library, the same operator can have different underlying implementations. For instance, vendors can provide multiple interfaces, and there are also various algorithms to choose from for handwriting. Therefore, for the same operator, how to select the backend with the best performance in different situations becomes a problem.

Type of Change

  • [√] feat — new feature / new operator / new platform
  • fix — bug fix
  • perf — performance improvement (no behavioral change)
  • refactor — code restructuring without behavior change
  • test — adding or fixing tests only
  • docs — documentation only
  • build / ci — build system or CI configuration
  • chore — tooling, formatting, or other non-code changes
  • Breaking change (requires a ! in the Conventional Commits prefix or a BREAKING CHANGE: footer)

Platforms Affected

  • CPU (WITH_CPU)
  • [√] NVIDIA (WITH_NVIDIA)
  • Iluvatar (WITH_ILUVATAR)
  • MetaX (WITH_METAX)
  • Cambricon (WITH_CAMBRICON)
  • Moore (WITH_MOORE)
  • Ascend (WITH_ASCEND)
  • PyTorch C++ bindings (WITH_TORCH)
  • Build system / CMake / CI
  • Python bindings / user-facing API

Smoke Test Result

root@master:~/InfiniOps# python -m pytest tests -m smoke -q
Running 75 items in this shard
........ss..ss....ss...............................ssssss..ss........ssssss                                                              [100%]
55 passed, 20 skipped, 18143 deselected in 43.80s

Test Results on Supported Platforms

Platform Affected Build / Smoke Result Full Result / Notes
NVIDIA smoke passed
Iluvatar
MetaX
Cambricon
Moore
Ascend
Full `pytest` output (optional)

Benchmark / Performance Impact

Notes for Reviewers

@mingdaw689
mingdaw689 requested a review from a team August 4, 2026 11:15
@mingdaw689 mingdaw689 changed the title auto-tuning system feat(auto-tuing): add a new auto-tuning system Aug 5, 2026
@voltjia
voltjia force-pushed the feat/auto-tuning branch 4 times, most recently from cd601eb to 3a041de Compare September 1, 2026 10:33
Comment thread src/operator.h
Comment on lines +89 to +91
if (!ListContains(dev_type, ActiveDevices<void>{})) {
return;
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个应该不需要吧?如果不是 active device 的话按理来说就不会启用和调用这个平台的运行时,而且哪怕真有这样的情况的话,这个行为就会静默返回而不是报错,感觉不应该是预期的行为。

Comment thread src/tuning.cc
Comment on lines +1 to +10
#include "tuning.h"

#include <algorithm>
#include <cctype>
#include <cstdlib>
#include <fstream>
#include <iostream>
#include <nlohmann/json.hpp>
#include <string_view>
#include <utility>

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

顺序应该需要调整吧?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants