Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
186 changes: 63 additions & 123 deletions 1_PREPARE/Methods/Research/FieldSHIFT-2/FieldSHIFT-2.md
Original file line number Diff line number Diff line change
@@ -1,129 +1,69 @@
# FieldSHIFT-2: Advanced Domain Shifting Framework
# FieldSHIFT-2: Domain Shifting Framework

FieldSHIFT-2 is a sophisticated system for executing domain shifting operations. It provides a high-level interface for users to transform input domains into shifted domains and generate corresponding dissertation outlines.
FieldSHIFT-2 contains research scripts for domain transformations,
dissertation/outline generation, evaluation, translation and analysis.
Real provider access and scientific-runtime qualification remain separate
from the existing-output preservation contract.

## Overview

FieldSHIFT-2 is a framework for domain shifting operations. It processes input domains, generates pro-shifted and shifted domains, and creates dissertation outlines. The system also includes translation and linguistic analysis features for result evaluation.

## Directory Structure
## Actual source layout

```text
FieldSHIFT-2/
├── src/
│ ├── core/
│ │ ├── FieldSHIFT-2.py
│ │ ├── config.py
│ │ └── utils.py
│ ├── modules/
│ │ ├── domain_processing/
│ │ │ ├── generate_pro_shift_domains.py
│ │ │ └── shift_domains.py
│ │ ├── dissertation/
│ │ │ ├── dissertation_outline_generator.py
│ │ │ ├── dissertation_generator.py
│ │ │ ├── dissertation_improver.py
│ │ │ └── dissertation_translation.py
│ │ └── analysis/
│ │ └── language_analysis.py
│ └── shared/
│ └── constants.py
├── tests/
│ └── [test files]
├── docs/
│ └── [documentation files]
├── config.py
├── utils.py
├── generate_pro_shift_domains.py
├── shift_domains.py
├── dissertation_outline_generator.py
├── dissertation_generator.py
├── dissertation_explainer.py
├── dissertation_improver.py
├── dissertation_translation.py
├── dissertation_entity_evaluation.py
├── dissertation_grant_evaluator.py
├── Meta_Analysis_Shifted_Dissertation.py
├── Meta_Analysis_Shifted_Dissertation_Outline.py
├── IN_PROGRESS/
│ ├── language_analysis.py
│ └── generate_pre_pro_shifted_grants.py
├── Inputs_and_Outputs/
│ ├── Domain/
│ ├── Pro-Shifted_Domain/
│ ├── Shifted_Domain/
│ ├── Shifted_Domain_Dissertation_Outline/
│ └── Analyses/
├── requirements.txt
├── README.md
└── .gitignore

## Key Components

### FieldSHIFT-2.py

- **Input**: Configuration file (specified via --config argument)
- **Output**: Manages and orchestrates the entire domain shifting process, writing results to all designated output folders.
- **Functionality**: Acts as the main entry point for the FieldSHIFT-2 framework, coordinating the execution of various modules and ensuring the workflow progresses smoothly from input to final output.

### generate_pro_shift_domains.py

- **Input**: Domain files from `Inputs_and_Outputs/Domain/`
- **Output**: Pro-shifted domain files in `Inputs_and_Outputs/Pro-Shifted_Domain/`
- **Functionality**: Processes raw domain files to generate pro-shifted domains, which serve as intermediate representations before the final domain shift. This step involves initial transformations and preparations for more advanced domain shifting.

### shift_domains.py

- **Input**: Pro-shifted domain files from `Inputs_and_Outputs/Pro-Shifted_Domain/`
- **Output**: Shifted domain files in `Inputs_and_Outputs/Shifted_Domain/`
- **Functionality**: Executes the core domain shifting operations, transforming pro-shifted domains into their final shifted versions. This module leverages advanced language models to perform sophisticated domain transformations.

### dissertation_outline_generator.py

- **Input**: Shifted domain files from `Inputs_and_Outputs/Shifted_Domain/`
- **Output**: Dissertation outline files in `Inputs_and_Outputs/Shifted_Domain_Dissertation_Outline/`
- **Functionality**: Generates structured dissertation outlines based on the shifted domain files, ensuring coherence and logical flow. This module creates a high-level framework for dissertations, organizing content into a clear and logical structure.

### dissertation_generator.py

- **Input**: Dissertation outline files from `Inputs_and_Outputs/Shifted_Domain_Dissertation_Outline/`
- **Output**: Complete dissertation drafts in `Inputs_and_Outputs/Shifted_Domain_Dissertation_Outline/`
- **Functionality**: Expands dissertation outlines into full drafts, adding detailed content and ensuring academic rigor. This module enhances the outlines with comprehensive sections, including introduction, methodology, results, and discussion.

### dissertation_improver.py

- **Input**: Dissertation drafts from `Inputs_and_Outputs/Shifted_Domain_Dissertation_Outline/`
- **Output**: Improved dissertation drafts in `Inputs_and_Outputs/Shifted_Domain_Dissertation_Outline/`
- **Functionality**: Enhances the quality of dissertation drafts by refining content, improving clarity, and ensuring adherence to academic standards. This module focuses on polishing the drafts to meet high academic and publication standards.

### dissertation_translation.py

- **Input**: Dissertation drafts from `Inputs_and_Outputs/Shifted_Domain_Dissertation_Outline/`
- **Output**: Translated dissertation drafts in `Inputs_and_Outputs/Shifted_Domain_Dissertation_Outline/`
- **Functionality**: Translates dissertation drafts into different languages, maintaining the original meaning and context. This module ensures that the dissertations are accessible to a broader audience by providing accurate translations.

### language_analysis.py

- **Input**: All folders in `Inputs_and_Outputs/`
- **Output**: Analysis results in `Inputs_and_Outputs/Analyses/`
- **Functionality**: Performs comprehensive linguistic analysis on all input and output files, providing insights into language patterns, quality, and consistency. This module evaluates the linguistic aspects of the content, ensuring high-quality and consistent language use across all documents.

### utils.py

- **Functionality**: Provides utility functions for file I/O, logging, and LLM setup. This module contains helper functions to streamline file operations, manage logs, and configure language models, ensuring smooth operation across the entire project. It is used by all other modules to perform common tasks efficiently.

### config.py

- **Input**: Configuration file specified by command-line argument
- **Output**: Configuration dictionary used by other modules
- **Functionality**: Loads and parses the configuration file, providing a configuration dictionary that other modules use to access settings and parameters. This module ensures that all components of the framework are correctly configured and can access necessary parameters.

## Installation

1. Clone the repository

## Usage

Run the scripts.

## Configuration

Create a Python configuration file specifying:

- Input/output directories
- Domain file paths
- Prompts
- LLM parameters
- Analysis parameters

## Contributing

Contribute on Github.
Contact <blanket@activeinference.institute>

## License
├── FieldSHIFT_Meta/
└── FieldSHIFT-1/
```

There is no `src/core/FieldSHIFT-2.py` master CLI or central `--config`
orchestrator. The scripts are flat modules except for the two named
`IN_PROGRESS` consumers. `config.py` supplies Python configuration; each
caller's actual directory/full-path arguments control its output, rather than
an invented common dissertation or analysis destination.
Preserved artifacts under `Inputs_and_Outputs/`, `FieldSHIFT_Meta/` and the
predecessor tree are not rewritten to match an aspirational layout.

## Writers and execution boundary

`save_file(content, full_path)` writes caller-serialized UTF-8 text; it has no
implicit directory/filename overload. Existing destinations, including empty
files, are preserved before artifact-specific input/provider/computation work.
Complete groups skip; partial groups preserve existing companions while
eligible missing outputs remain independent. Both meta-analysis modules
participate in this contract.

`IN_PROGRESS/language_analysis.py` writes `analysis.json` and a conditional
PCA companion. Existing companions are preserved independently; nested
linguistic features are not silently flattened into a new scientific model.

Normal imports and entrypoints retain their optional-library, key/configuration
and NLTK prerequisites. Preservation controls are not credential-free execution,
provider integration or permission to generate research outputs. Grant-product
generation and all-folder language analysis remain separate owner decisions;
see [the original pending requests](IN_PROGRESS/FieldShift_TODO.md).

## References and maintenance

The [pipeline inventory](README.md) and [writer specification](SPEC.md) describe
the current local interfaces. The predecessor paper and its historical CSVs
remain historical evidence, not newly executed writer entrypoints.

Contribute through the repository's normal review process.
Contact <blanket@activeinference.institute>.
Repository license: CC BY-NC-ND 4.0.

...
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
Check all methods that they skip files already there.
Existing-file preservation: the complete writer/caller ledger, both nested
scripts, both meta-analysis modules, empty-file preservation, partial groups
and offline no-provider-work cases passed the parent's governed focused
software acceptance on 2026-11-13. This closure is not based on only four
clustering files and is not credential/import-free entrypoint or scientific
runtime acceptance. The grant-generation and all-folder language-analysis
requests below remain separate and open (FS-2/FS-3).

Write Pro-Grant (Completed catechism) for the Funding calls in the Grant Fond, for funding that PhD student.

Expand Down
7 changes: 7 additions & 0 deletions 1_PREPARE/Methods/Research/FieldSHIFT-2/IN_PROGRESS/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,13 @@ IN PROGRESS — component of the FieldSHIFT-2 subsystem in the Active InferAnts
- 📄 `generate_pre_pro_shifted_grants.py`
- 📄 `language_analysis.py`

## Companion outputs

Linguistic output comprises `analysis.json` and a conditional PCA image.
Existing companions are preserved independently, including empty files.
Default nested linguistic features are not silently flattened into a new
numeric feature model; scientific feature/runtime selection remains separate.

## Related

- Parent: [FieldSHIFT-2](../README.md)
7 changes: 7 additions & 0 deletions 1_PREPARE/Methods/Research/FieldSHIFT-2/IN_PROGRESS/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,13 @@ IN PROGRESS — component of the FieldSHIFT-2 subsystem in the Active InferAnts
- 📄 `generate_pre_pro_shifted_grants.py`
- 📄 `language_analysis.py`

## Companion contract

Linguistic output comprises `analysis.json` and a conditional PCA image.
Existing companions are preserved independently. Default nested linguistic
features are not silently flattened into a new numeric feature model;
scientific feature/runtime selection remains separate.

## Version

- **Framework**: Active InferAnts
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,14 +51,18 @@ def generate_concatenated_domains(domains: List[str], prompt: str, output_dir: s
for domain_a, domain_b in product(domains, repeat=2):
domain_a_name = os.path.splitext(os.path.basename(domain_a))[0]
domain_b_name = os.path.splitext(os.path.basename(domain_b))[0]
output_filename = f'concatenated_{domain_a_name}_to_{domain_b_name}.md'
output_path = os.path.join(output_dir, output_filename)
if os.path.exists(output_path):
logger.info(f"Output already exists; preserving {output_path}")
continue
logger.info(f"Processing combination: {domain_a_name} and {domain_b_name}")
domain_a_content = load_file(domain_a)
domain_b_content = load_file(domain_b)

concatenated_content = generate_pro_shift_domains(domain_a_content, domain_b_content, prompt)

output_filename = f'concatenated_{domain_a_name}_to_{domain_b_name}.md'
save_file(concatenated_content, output_dir, output_filename)
save_file(concatenated_content, output_path)

logger.info(f"Concatenated domain saved as: {output_filename}")
completed_combinations += 1
Expand Down Expand Up @@ -88,21 +92,25 @@ def process_domains(config: dict = DEFAULT_CONFIG) -> None:

# Load domain contents
domain_files = [os.path.join(input_domain_dir, f) for f in os.listdir(input_domain_dir) if f.endswith('.md')]
domains = [load_file(file) for file in domain_files]

logger.info("Domains loaded successfully")
logger.info("Starting concatenation process")

# Load the prompt
prompt_path = os.path.join(base_dir, 'Prompts', 'FieldSHIFT-2-Domain_Shift.md')
prompt = load_file(prompt_path)

# Generate concatenated domains
generate_concatenated_domains(domain_files, prompt, config['output_dirs']['Concatenated_Domain'])
output_dir = config['output_dirs']['Concatenated_Domain']
if any(
not os.path.exists(os.path.join(
output_dir,
f"concatenated_{os.path.splitext(os.path.basename(a))[0]}_to_{os.path.splitext(os.path.basename(b))[0]}.md",
))
for a, b in product(domain_files, repeat=2)
):
prompt_path = os.path.join(base_dir, 'Prompts', 'FieldSHIFT-2-Domain_Shift.md')
prompt = load_file(prompt_path)
generate_concatenated_domains(domain_files, prompt, output_dir)

logger.info("Concatenation process completed successfully.")
print(f"\n{'='*50}\nConcatenation process summary:\n{'-'*50}")
print(f"Total domains processed: {len(domains)}")
print(f"Total domains processed: {len(domain_files)}")
print(f"Output directory: {config['output_dirs']['Concatenated_Domain']}")
print(f"{'='*50}\n")
except Exception as e:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import os
import json
import logging
from typing import List, Dict, Any
from textstat import textstat
Expand Down Expand Up @@ -34,6 +35,9 @@ def calculate_readability(text: str) -> Dict[str, float]:
}

def perform_pca_analysis(features: List[float], output_dir: str, filename: str) -> None:
pca_plot_path = os.path.join(output_dir, f'{filename}_pca.png')
if os.path.exists(pca_plot_path):
return
logger.info(f"Performing PCA analysis for {filename}")
pca = PCA(n_components=2)
principal_components = pca.fit_transform(features)
Expand All @@ -44,7 +48,6 @@ def perform_pca_analysis(features: List[float], output_dir: str, filename: str)
plt.xlabel('Principal Component 1')
plt.ylabel('Principal Component 2')

pca_plot_path = os.path.join(output_dir, f'{filename}_pca.png')
plt.savefig(pca_plot_path)
logger.info(f"PCA plot saved to {pca_plot_path}")

Expand All @@ -68,6 +71,10 @@ def analyze_text(text: str, analysis_params: Dict[str, Any]) -> Dict[str, Any]:
return results

def perform_linguistic_analysis(input_dir: str, output_dir: str) -> None:
analysis_path = os.path.join(output_dir, 'analysis.json')
pca_path = os.path.join(output_dir, f'{os.path.basename(input_dir)}_pca.png')
if os.path.exists(analysis_path) and os.path.exists(pca_path):
return
analysis_params = DEFAULT_CONFIG['analysis_params']
results = {}
all_features = []
Expand All @@ -83,10 +90,11 @@ def perform_linguistic_analysis(input_dir: str, output_dir: str) -> None:
all_features.append(list(analysis_result.values()))

os.makedirs(output_dir, exist_ok=True)
save_file(results, output_dir, 'analysis.json')
logger.info(f"Analysis results saved to {os.path.join(output_dir, 'analysis.json')}")
if not os.path.exists(analysis_path):
save_file(json.dumps(results, indent=2), analysis_path)
logger.info(f"Analysis results saved to {analysis_path}")

if all_features and all_features[0]:
if not os.path.exists(pca_path) and all_features and all_features[0]:
perform_pca_analysis(all_features, output_dir, os.path.basename(input_dir))

def perform_meta_analysis(base_dir: str) -> None:
Expand Down
Loading
Loading