# changelog

author: Tim Theisen <tim@cs.wisc.edu> - 23.0.8-1
changelog:
  - Fix bug where ssh-agent processes were leaked with grid universe jobs
  - Fix DAGMan crash when a provisioner node was given a parent
  - Fix bug that prevented use of "ftp:" URLs in file transfer
  - Fix bug where jobs that matched an offline slot never start


author: Tim Theisen <tim@cs.wisc.edu> - 23.6.1-1
changelog:
  - Add the ability to force vanilla universe jobs to run in a container
  - Add the ability to override the entrypoint for a Docker image
  - condor_q -better-analyze includes units for memory and disk quantities


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.10-1
changelog:
  - Preliminary support for Ubuntu 22.04 (Noble Numbat)
  - Warns about deprecated multiple queue statements in a submit file
  - Fix bug where plugins could not signify to retry a file transfer
  - The condor_upgrade_check script checks for proper token file permissions
  - Fix bug where the condor_upgrade_check script crashes on older platforms
  - The bundled version of apptainer was moved to libexec in the tarball


author: Tim Theisen <tim@cs.wisc.edu> - 23.7.2-1
changelog:
  - Warns about deprecated multiple queue statements in a submit file
  - The semantics of 'skip_if_dataflow' have been improved
  - Removing large DAGs is now non-blocking, preserving schedd performance
  - Periodic policy expressions are now checked during input file transfer
  - Local universe jobs can now specify a container image
  - File transfer plugins can now advertise extra attributes
  - DAGMan can rescue and abort if pending jobs are missing from the job queue
  - Fix so 'condor_submit -interactive' works on cgroup v2 execution points


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.12-1
changelog:
  - Remote condor_history queries now work the same as local queries
  - Improve error handling when submitting to a remote scheduler via ssh
  - Fix bug on Windows where condor_procd may crash when suspending a job
  - Fix Python binding crash when submitting a DAG which has empty lines


author: Tim Theisen <tim@cs.wisc.edu> - 23.8.1-1
changelog:
  - Add new condor-ap package to facilitate Access Point installation
  - HTCondor Docker images are now based on Alma Linux 9
  - HTCondor Docker images are now available for the arm64 CPU architecture
  - The user can now choose which submit method DAGMan will use
  - Can add custom attributes to the User ClassAd with condor_qusers -edit
  - Add use-projection option to condor_gangliad to reduce memory footprint
  - Fix bug where interactive submit does not work on cgroup v2 systems (EL9)


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.14-1
changelog:
  - Docker and Container jobs run on EPs that match AP's CPU architecture
  - Fixed premature cleanup of credentials by the condor_credd
  - Fixed bug where a malformed SciToken could cause a condor_schedd crash
  - Fixed crash in condor_annex script
  - Fixed daemon crash after IDTOKEN request is approved by the collector


author: Tim Theisen <tim@cs.wisc.edu> - 23.9.6-1
changelog:
  - Add config knob to not have cgroups count kernel memory for jobs on EL9
  - Remove support for numeric unit suffixes (k,M,G) in ClassAd expressions
  - In submit files, request_disk & request_memory still accept unit suffixes
  - Hide GPUs not allocated to the job on cgroup v2 systems such as EL9
  - DAGMan can now produce credentials when using direct submission
  - Singularity jobs have a contained home directory when file transfer is on
  - Avoid using IPv6 link local addresses when resolving hostname to IP addr
  - New 'htcondor credential' command to aid in debugging


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.15-1
changelog:
  - Fix bug where Docker universe jobs reported zero memory usage on EL9
  - Fix bug where Docker universe images would not be removed from EP cache
  - Fix bug where condor_watch_q could crash
  - Fix bug that could cause the file transfer hold reason to be truncated
  - Fix bug where a Windows job with a bad executable would not go on hold


author: Tim Theisen <tim@cs.wisc.edu> - 23.10.1-1
changelog:
  - Improvements to disk usage enforcement when using LVM
    - Can encrypt job sandboxes when using LVM
    - More precise tracking of disk usage when using LVM
    - Reduced disk usage tracking overhead
  - Improvements tracking CPU and memory usage with cgroup v2 (on EL9)
    - Don't count kernel cache pages against job's memory usage
    - Avoid rare inclusion of previous job's CPU and peak memory usage
  - HTCondor now re-checks DNS before re-connecting to a collector
  - HTCondor now writes out per job epoch history
  - HTCondor can encrypt network connections without requiring authentication
  - htcondor CLI can now show status for local server, AP, and CM
  - htcondor CLI can now display OAUTH2 credentials
  - Uses job's sandbox to convert image format for Singularity/Apptainer
  - Bug fix to not lose GPUs in Docker job on systemd reconfig
  - Bug fix for PID namespaces and condor_ssh_to_job on EL9


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.17-1
changelog:
  - Bug fix for PID namespaces and condor_ssh_to_job on EL9
  - Augment condor_upgrade_check to find unit suffixes in ClassAd expressions


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.18-1
changelog:
  - Proper error message and hold when Docker emits multi-line error message
  - The htcondor CLI now works on Windows


author: Tim Theisen <tim@cs.wisc.edu> - 10.0.8-1
changelog:
  - Avoid kernel panic on some Enterprise Linux 8 systems
  - Fix bug where early termination of service nodes could crash DAGMan
  - Limit email about long file transfer queue to once daily
  - Various fixes to condor_adstash


author: Tim Theisen <tim@cs.wisc.edu> - 10.8.0-1
changelog:
  - Fold the classads, blahp, and procd RPMs into the main condor RPM
  - Align the Debian packages and package names with the RPM packaging
  - On Linux, the default configuration enforces memory limits with cgroups
  - condor_status -gpus shows nodes with GPUs and the GPU properties
  - condor_status -compact shows a row for each slot type
  - New ENV command controls which environment variables are present in DAGMan


author: Tim Theisen <tim@cs.wisc.edu> - 10.0.9-1
changelog:
  - The condor_upgrade_check script now provides guidance on updating to 23.0
  - The htchirp Python binding now properly locates the chirp configuration
  - Fix bug that prevented deletion of HTCondor passwords on Windows


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.0-1
changelog:
  - Absent slot configuration, execution points will use a partitionable slot
  - Linux cgroups enforce maximum memory utilization by default
  - Can now define DAGMan save points to be able to rerun DAGs from there
  - Much better control over environment variables when using DAGMan
  - Administrators can enable and disable job submission for a specific user
  - Can set a minimum number of CPUs allocated to a user
  - condor_status -gpus shows nodes with GPUs and the GPU properties
  - condor_status -compact shows a row for each slot type
  - Container images may now be transferred via a file transfer plugin
  - Support for Enterprise Linux 9, Amazon Linux 2023, and Debian 12
  - Can write job information in AP history file for every execution attempt
  - Can run defrag daemons with different policies on distinct sets of nodes
  - Add condor_test_token tool to generate a short lived SciToken for testing
  - The job’s executable is no longer renamed to ‘condor_exec.exe’


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.1-1
changelog:
  - Fix 10.6.0 bug that broke PID namespaces
  - Fix bug where execution times for ARC CE jobs were 60 times too large
  - Fix bug where a failed 'Service' node would crash DAGMan
  - Condor-C and Job Router jobs now get resources provisioned updates


author: Tim Theisen <tim@cs.wisc.edu> - 23.1.0-1
changelog:
  - Enhanced filtering with 'condor_watch_q'
  - Can specify alternate ssh port with 'condor_remote_cluster'
  - Performance improvement for the 'condor_schedd' and other daemons
  - Jobs running on cgroup v2 systems can subdivide their cgroup
  - The curl plugin can now find CA certificates via an environment variable


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.2-1
changelog:
  - Fix bug where OIDC login information was missing when submitting jobs
  - Improved sandbox and ssh-agent clean up for batch grid universe jobs
  - Fix bug where daemons with a private network address couldn't communicate
  - Fix cgroup v2 memory enforcement for custom configurations
  - Add DISABLE_SWAP_FOR_JOB support on cgroup v2 systems
  - Fix log rotation for OAuth and Vault credmon daemons


author: Tim Theisen <tim@cs.wisc.edu> - 23.2.0-1
changelog:
  - Add 'periodic_vacate' submit command to restart jobs that are stuck
  - EPs now advertises whether the execute directory is on rotational storage
  - Add two log events for the time a job was running and occupied a slot
  - Files written by HTCondor are now written in binary mode on Windows
  - HTCondor now uses the Pelican Platform for OSDF file transfers


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.3-1
changelog:
  - Preliminary support for openSUSE LEAP 15
  - All non-zero exit values from file transfer plugins are now errors
  - Fix crash in Python bindings when job submission fails
  - Chirp uses a 5120 byte buffer and errors out for bigger messages
  - condor_adstash now recognizes GPU usage values as floating point numbers


author: Tim Theisen <tim@cs.wisc.edu> - 23.3.0-1
changelog:
  - Restore limited support for Enterprise Linux 7 systems
  - Additional assistance converting old syntax job routes to new syntax
  - Able to capture output to debug DAGMan PRE and POST scripts
  - Execution Points advertise when jobs are running with cgroup enforcement


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.4-1
changelog:
  - NVIDIA_VISIBLE_DEVICES environment variable lists full uuid of slot GPUs
  - Fix problem where some container jobs would see GPUs not assigned to them
  - Restore condor keyboard monitoring that was broken since HTCondor 23.0.0
  - In condor_adstash, the search engine timeouts now apply to all operations
  - Ensure the prerequisite perl modules are installed for condor_gather_info


author: Tim Theisen <tim@cs.wisc.edu> - 23.4.0-1
changelog:
  - condor_submit warns about unit-less request_disk and request_memory
  - Separate condor-credmon-local RPM package provides local SciTokens issuer
  - Fix bug where NEGOTIATOR_SLOT_CONSTRAINT was ignored since version 23.3.0
  - The htcondor command line tool can process multiple event logs at once
  - Prevent Docker daemon from keeping a duplicate copy of the job's stdout


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.6-1
changelog:
  - Fix DAGMan where descendants of removed retry-able jobs are marked futile
  - Ensure the condor_test_token works correctly when invoked as root
  - Fix bug where empty multi-line values could cause a crash
  - condor_qusers returns proper exit code for errors in formatting options
  - Fix crash in job router when a job transform is missing an argument


author: Tim Theisen <tim@cs.wisc.edu> - 23.5.2-1
changelog:
  - Old ClassAd based syntax is disabled by default for the job router
  - Can efficiently manage/enforce disk space using LVM partitions
  - GPU discovery is enabled on all Execution Points by default
  - Prevents accessing unallocated GPUs using cgroup v1 enforcement
  - New condor_submit commands for constraining GPU properties
  - Add ability to transfer EP's starter log back to the Access Point
  - Can use VOMS attributes when mapping identities of SSL connections
  - The CondorVersion string contains the source git SHA


author: Tim Theisen <tim@cs.wisc.edu> - 23.7.1-0.1
changelog:
  - Warns about deprecated multiple queue statements in a submit file
  - The semantics of 'skip_if_dataflow' have been improved
  - Removing large DAGs is now non-blocking, preserving schedd performance
  - Periodic policy expressions are now checked during input file transfer
  - Local universe jobs can now specify a container image
  - File transfer plugins can now advertise extra attributes
  - DAGMan can rescue and abort if pending jobs are missing from the job queue


author: Tim Theisen <tim@cs.wisc.edu> - 5.1.4-1
changelog:
  - Fix whole node job glidein CPUs and GPUs expressions that caused held jobs
  - Fix bug where default CERequirements were being ignored
  - Pass whole node request from GlideinWMS to the batch system
  - Since CentOS 8 has reached end of life, we build and test on Rocky Linux 8


author: Tim Theisen <tim@cs.wisc.edu> - 5.1.5-1
changelog:
  - Rename AuthToken attributes in the routed job to better support accounting
  - Prevent GSI environment from pointing the job to the wrong certificates
  - Fix issue where HTCondor-CE would need port 9618 open to start up


author: Tim Theisen <tim@cs.wisc.edu> - 5.1.6-1
changelog:
  - HTCondor-CE now uses the C++ Collector plugin for payload job traceability
  - Fix HTCondor-CE mapfiles to be compliant with PCRE2 and HTCondor 9.10.0+
  - Add support for multiple APEL accounting scaling factors
  - Suppress spurious log message about a missing negotiator
  - Fix crash in HTCondor-CE View


author: Tim Theisen <tim@cs.wisc.edu> - 6.0.0-1
changelog:
  - Align HTCondor-CE security configuration with HTCondor defaults
  - Add example configuration on how to ban users
  - Add condor_ce_transform_ads command
  - Improve essential directory checking and creation at startup


author: Tim Theisen <tim@cs.wisc.edu> - 6.0.1-1
changelog:
  - Add grid CA and host certificate/key locations to default SSL search paths
  - Verifies that HTCondor-CE can access the local HTCondor's SPOOL directory
  - Can use condor_ce_trace without SciToken to test batch system integration
  - condor_ce_upgrade_check checks compatibility with next HTCondor version
  - Adds deprecation warnings for old job router configuration syntax


author: Tim Theisen <tim@cs.wisc.edu> - 23.0.5-1
changelog:
  - Fix CE job route transform for job environment
  - Fix CERequirements when the default_CERequirements is not set
  - Add condor_ce_test_token tool to generate short lived SciToken for tests
  - Remove GSI from security method list to eliminate annoying warnings


author: Tim Theisen <tim@cs.wisc.edu> - 23.9.1-1
changelog:
  - Use new Job Router syntax by default
  - Update configuration files to work with HTCondor 23.9.1 and later


