Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync 231206 #2321

Merged
merged 397 commits into from
Dec 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
397 commits
Select commit Hold shift + click to select a range
ac14d91
Wrap line to 80 characters
mihaimaruseac Nov 28, 2023
2acd3c1
Merge pull request #62402 from tensorflow:sushreebarsa-patch-2
tensorflower-gardener Nov 28, 2023
af0eaf0
Fix duplicate checkpoint removal in Saver class
proxOP Nov 28, 2023
5f3f90d
Merge branch 'master' of https://github.com/tensorflow/tensorflow
proxOP Nov 28, 2023
6880b6a
Update XNNPACK and cpuinfo version
tensorflower-gardener Nov 28, 2023
017e96a
PR #7323: [ROCm] Restore private visibility of stream executor privat…
i-chaochen Nov 28, 2023
051bde1
[XLA:CPU] Refactor local collectives into a separate file behind an i…
hawkinsp Nov 28, 2023
ee10e3d
[XLA:GPU][NFC] Split GemmRewriterTriton into 4 parts
tdanyluk Nov 28, 2023
247f77e
Add type annotations to test_util.py.
Spyboticsguy Nov 28, 2023
8407c14
Failed compatibility tests were disabled for older version plugins an…
Nov 28, 2023
8d84d04
`third_party/gpus` changes from PR #7277 that were missed
ddunl Nov 28, 2023
1436d4d
Rewrite debugger functionalities in c++.
dansuh17 Nov 28, 2023
44ac2ac
Add missing #includes that define symbols referenced by simple_*deleg…
liangyaning33 Nov 28, 2023
3c9304a
Make the allocated memory array size private, only allow querying the…
lrdxgm Nov 28, 2023
ddc4140
Remove a dangling TODO comment.
abattery Nov 28, 2023
37f9708
[xla:runtime] NFC: Remove runner library
ezhulenev Nov 29, 2023
af3d28d
Enable Linux Arm64 GitHub presubmit
nitins17 Nov 29, 2023
87959ee
PR #7136: [XLA:GPU] Add `Allocate` command to command buffer
shawnwang18 Nov 29, 2023
14f8066
Don't upload macOS Arm64 build artifacts.
carlos-guia Nov 29, 2023
4f1d4bc
Update XLA GPU config with NVCC compiler.
tensorflower-gardener Nov 29, 2023
8509ee8
For convolutions that can be interpreted as dots, rely on DotHandler …
tensorflower-gardener Nov 29, 2023
192718f
[stream_executor] Record cond_builder before evaluating loop conditio…
ezhulenev Nov 29, 2023
b67b97e
Implemented QuantizeConvolutionOpPattern and integration test, integr…
tensorflower-gardener Nov 29, 2023
fc8a13b
[stream_executor] Add Memset command to CommandBuffer
ezhulenev Nov 29, 2023
16f2028
[stream_executor] Add support for updating Memcpy command parameters
ezhulenev Nov 29, 2023
739d18a
Converts the AutoShardingSolverRequest object into a proto (which pro…
tensorflower-gardener Nov 29, 2023
ec89cbd
Merge branch 'tensorflow:master' into master
proxOP Nov 29, 2023
0d829d7
Integrate StableHLO at openxla/stablehlo@83f095e7
ghpvnist Nov 29, 2023
f0240ad
Re-factor macOS CI build environment setup
nitins17 Nov 29, 2023
e1dbfeb
[stream_executor] NFC: Guard new features with CUDA_VERSION check
ezhulenev Nov 29, 2023
f9a4e14
[XLA] Add support for while loop simplifier to remove duplicated "dyn…
Nov 29, 2023
cfabb0d
Merge pull request #62405 from elfringham:no_fork_osv_scan
tensorflower-gardener Nov 29, 2023
5b98c79
[XLA:GPU] Add a pass merging producer fusions in to Triton Softmax fu…
jvstokes Nov 29, 2023
941c1a1
[XLA] Fix the MHLO TopK `assemblyFormat` to correctly support the opt…
dimitar-asenov Nov 29, 2023
8a283df
Update GraphDef version to 1695.
tensorflower-gardener Nov 29, 2023
dedcf06
compat: Update forward compatibility horizon to 2023-11-29
tensorflower-gardener Nov 29, 2023
7a7a4c1
[XLA:GPU][NFC] Beautify code related to Triton fusions
tdanyluk Nov 29, 2023
1403549
Fix a compile error caused by in_process_collectives
apaszke Nov 29, 2023
71f59bb
[TileAnalysis] Add indexing computation for reshape.
pifon2a Nov 29, 2023
34c440c
Fix invalid conditional check in macOS CI script
nitins17 Nov 29, 2023
7486607
Basic simplifier for indexing maps.
jreiffers Nov 29, 2023
56016f2
[TileAnalysis] Add indexing computation for HloBitcast.
pifon2a Nov 29, 2023
fe25113
[XLA:GPU] Append ".0" suffix to all instructions names.
olegshyshkov Nov 29, 2023
13cd409
[xla:gpu] NFC: Migrate custom CUTLASS kernel to GemmUniversal templates
ezhulenev Nov 29, 2023
2c822d6
[XLA:GPU][NFC] Refactor tiling propagation.
Nov 29, 2023
cc7d75a
[XLA] Lower verbosity in MSA.
berkinilbeyi Nov 29, 2023
e19035f
Record free_gpu_system_memory using addressable_devices() to avoid po…
tensorflower-gardener Nov 29, 2023
755370c
Limit "single source file per target" presubmit to check only python …
jam14j Nov 29, 2023
890ed36
Integrate LLVM at llvm/llvm-project@3287ae8f6520
gribozavr Nov 29, 2023
23b87eb
[XLA:CPU] Add a direct implementation of AllGather, rather than lower…
hawkinsp Nov 29, 2023
4d64829
User sharding annotations can sometimes be invalid wrt to the shape o…
tensorflower-gardener Nov 29, 2023
cc27487
Some cleanup: Inline a function used once, remove some dead code, and…
tensorflower-gardener Nov 29, 2023
7d4df89
Reenable `BitcastConvert` tests in JAX CI now that the tests pass again
ddunl Nov 29, 2023
7f89ef6
Update TFRT dependency to use revision
tensorflower-gardener Nov 29, 2023
2d6d24d
Adds the ability to dump solver request protos.
tensorflower-gardener Nov 29, 2023
78ca91b
Skip two tests in tensorflow/python/module:module_test, on Python 3.12.
tensorflower-gardener Nov 29, 2023
2542008
Fastpath for setting disable-jit.
pschuh Nov 29, 2023
fb54a86
When iterative solving is turned on, shardings annotations from a pre…
tensorflower-gardener Nov 29, 2023
8ec0413
Move large functions in auto_sharding_dot_handler.cc out of class def…
tensorflower-gardener Nov 29, 2023
812596b
[XLA]
tensorflower-gardener Nov 29, 2023
7aeb9c8
Changed Version of Bazel to version 6.4.0
tensorflower-gardener Nov 29, 2023
0749e6a
[XLA:CPU] Add a direct implementation of ReduceScatter, instead of lo…
hawkinsp Nov 29, 2023
bf163a7
PR #7370: [XLA:GPU] fix command_buffer_thunk_test failure
shawnwang18 Nov 29, 2023
d9e9f33
[xla:ffi] Added error reporting to existing decoders
superbobry Nov 29, 2023
e2b224e
[XLA:CPU] Add a direct implementation of ReduceScatter, instead of lo…
hawkinsp Nov 29, 2023
b0cc761
Remove obselete TODO
ddunl Nov 29, 2023
3a5c401
[XLA:CPU] Add a direct implementation of AllGather, rather than lower…
hawkinsp Nov 29, 2023
73f5d2f
Remove tensorflow namespace from `tsl/platform/status_matchers.h`
ddunl Nov 29, 2023
ba9f661
[XLA:LatencyHidingScheduler] Prefer picking the instruction with less…
tensorflower-gardener Nov 29, 2023
9da5526
Remove math_ops.py's indirect dependency on resource_variable_ops.py …
fionalang Nov 29, 2023
e0693f7
[xla:ffi] Parameterized ffi::BaseBuffer with its rank
superbobry Nov 29, 2023
44a6a97
[XLA] Allow moving for HloSharding in xla::HloInstruction::set_sharding.
tensorflower-gardener Nov 29, 2023
7b451a1
Add EvalOrPattern to StablehloRefineShapes pass
tensorflower-gardener Nov 29, 2023
bfed4a5
[XLA] Allow tuple_shardings move in the HloSharding ctor.
tensorflower-gardener Nov 29, 2023
5759fac
[xla:gpu] Move external allocation implementation details from Stream…
ezhulenev Nov 29, 2023
2b0099b
[XLA] Do not create temporary vector in HloSharding::GetSubSharding.
tensorflower-gardener Nov 29, 2023
cbd7da3
[XLA] Remove an unnecessary HloSharding assignment.
tensorflower-gardener Nov 30, 2023
474f447
Use clang+NVCC compilers for all XLA GPU jobs.
tensorflower-gardener Nov 30, 2023
413a3d4
[XLA] Move HloSharding where possible.
tensorflower-gardener Nov 30, 2023
a0f9e8b
Remove `alwayslink = True` for stablehlo bridge passes.
dansuh17 Nov 30, 2023
467205f
[XLA] Change the prototype of ReturnImprovedShardingImpl to minmize c…
tensorflower-gardener Nov 30, 2023
c8ec05f
[XLA] Optimize HloValue::ComputeUses().
tensorflower-gardener Nov 30, 2023
c7e84a2
[xla:gpu] Own command buffer allocations at a Thunk level
ezhulenev Nov 30, 2023
5dd7256
Integrate LLVM at llvm/llvm-project@f688e0901213
tensorflower-gardener Nov 30, 2023
a492d58
[stream_executor] Use new CUDA runtime API for TopK
tyb0807 Nov 30, 2023
961ca34
[xla:gpu] Add the AOT compilation pipeline for thunk runtime #7360
anlunx Nov 30, 2023
ca6672c
Update TFRT dependency to use revision
tensorflower-gardener Nov 30, 2023
086f52f
[XLA:GPU] Improve errors if callbacks are not provided to the Collect…
hawkinsp Nov 30, 2023
49184b6
Relocates the evaluation output into the core solver.
tensorflower-gardener Nov 30, 2023
a450666
Lower tensor.from_elements and shape.broadcast ops in ShapeLegalizeToHLO
tensorflower-gardener Nov 30, 2023
f85e22b
Internal infrastructure change
GleasonK Nov 30, 2023
7f0aff4
[stream_executor][NFC] More doc for While and For command
tyb0807 Nov 30, 2023
8e4e31e
Merge pull request #62496 from proxOP:master
tensorflower-gardener Nov 30, 2023
3149368
Update GraphDef version to 1696.
tensorflower-gardener Nov 30, 2023
67a62c4
compat: Update forward compatibility horizon to 2023-11-30
tensorflower-gardener Nov 30, 2023
5d2c0b1
Slightly improve power of 2 rounding in TreeReductionEmitter.
akuegel Nov 30, 2023
1b83acb
[XLA] [NFC] Minor HTML rendering QoL improvements
cheshire Nov 30, 2023
a8d0005
Add support for simplifying reverse.
jreiffers Nov 30, 2023
f2e5769
[TileAnalysis] Add an example for a fused dot.
pifon2a Nov 30, 2023
52c26c6
[XLA:GPU] Don't look for roots in non-fusion computation.
olegshyshkov Nov 30, 2023
7ce56ce
Fix reduce-reduce check.
jreiffers Nov 30, 2023
ce2a8cf
Merge pull request #62299 from Intel-tensorflow:amin/jax-regression
tensorflower-gardener Nov 30, 2023
33f9004
Integrate LLVM at llvm/llvm-project@511ba45a47d6
gribozavr Nov 30, 2023
43c17d7
[XLA] New interactive HTML visualization mode, available through hlo-opt
cheshire Nov 30, 2023
5cdd8ee
Update TFRT dependency to use revision
tensorflower-gardener Nov 30, 2023
882f1e1
[xla:gpu] Split CUTLASS gemm kernel compilation into a separate target
ezhulenev Nov 30, 2023
0d8a6a4
[XLA:GPU] Tiling propagation: generalize handling of concatenations.
Nov 30, 2023
45374d5
Add overload of `ModifyGraphWithDelegate` that takes a `TfLiteOpaqueD…
fergushenderson Nov 30, 2023
e4a3135
Disable map_fusion experiment due to suspected errors.
aaudiber Nov 30, 2023
fb55ba3
Remove `alwayslink = True` for the target `quantization/stablehlo:pas…
dansuh17 Nov 30, 2023
297e7ec
[XLA:CPU] Enforce a major-to-minor layout constraint on the TopK cust…
hawkinsp Nov 30, 2023
dc6be35
[stream_executor][NFC] Remove unused SharedDeviceMemory
tyb0807 Nov 30, 2023
14855f5
[XLA:CPU] Add a direct implementation of AllGather, rather than lower…
hawkinsp Nov 30, 2023
cbf5fef
Import openai/triton from GitHub.
Nov 30, 2023
442bf89
Moves the BuildStrategyAndCost() method into its own file (auto_shard…
tensorflower-gardener Nov 30, 2023
d2a94e9
[stream_executor][NFC] Cleanup topk_kernel refactor
tyb0807 Nov 30, 2023
fec118a
[xla:ffi] Split BufferBase into a weakly typed BufferBase and a templ…
superbobry Nov 30, 2023
14b2403
[HloValueSemanticsAnalysis] Handle OptimizationBarrier.
jinliangwei Nov 30, 2023
2b918e2
No public description
jimlinntu Nov 30, 2023
20c0589
[XLA:GPU] Fix fusion parameter limit again and remove hard check
tdanyluk Nov 30, 2023
870c7fa
#tf-data Add `buffer_output_elements` and `prefetch_input_elements` t…
wilsingosti Nov 30, 2023
d0366fc
Import ref_variable.py inline in variable_v1.py to ensure the tf1 var…
fionalang Nov 30, 2023
eba052c
Modifies the return type of `WriteToString` to include whether it's s…
tensorflower-gardener Nov 30, 2023
77ffe8b
#tensorflow enable double-typed gauge cell
jimlinntu Nov 30, 2023
73ac629
Adds logs statements to report the amount of time spent in the constr…
tensorflower-gardener Nov 30, 2023
2f0ff66
Import resource_variable_ops.py inline in variables.py to ensure the …
fionalang Nov 30, 2023
2efd24e
Turn static functions into regular free functions in `lift_quantizabl…
dansuh17 Nov 30, 2023
f8c92f3
Stop throwing exceptions if users pass `profiler_outdir` to `trace_ex…
tensorflower-gardener Nov 30, 2023
69c72ef
Add a way to serialize CompiledMemoryStats.
krishnaharidasan Dec 1, 2023
595f5ea
[tflite] Fix tflite selective_build_script.
grantjensen Dec 1, 2023
7378f1e
Add avg time in op detail
lionelfeng Dec 1, 2023
c655e08
Rework ShardingCallback to be more user-friendly.
BlaziusMaximus Dec 1, 2023
cff48a3
Add comments to the CallORToolsSolver to specify what fields of the r…
tensorflower-gardener Dec 1, 2023
c810fe7
[xla:gpu] Add support for control flow command buffer commands to Thunk
ezhulenev Dec 1, 2023
3a029b1
[HloValueSemanticsAnalysis] Use node_hash_map instead since the curre…
jinliangwei Dec 1, 2023
8683874
PR #7106: Pre-prepare profiler annotations for kernels/thunks and mod…
olupton Dec 1, 2023
cb792e6
[stream_executor] Put CUDA C++ kernels under if_cuda guard
ezhulenev Dec 1, 2023
f39fcef
Update GraphDef version to 1697.
tensorflower-gardener Dec 1, 2023
db58265
compat: Update forward compatibility horizon to 2023-12-01
tensorflower-gardener Dec 1, 2023
54af26e
Refactor fused quantization patterns into quantize pass.
chococigar Dec 1, 2023
75571c6
[xla:gpu] Add a flag to tell ir_emitter to not emit LLVM kernels #7360
anlunx Dec 1, 2023
f38c8ca
[XLA:GPU] Handle further propagation from a trivial sized tensor grac…
tdanyluk Dec 1, 2023
9983fe4
[XLA:CPU] Add a direct implementation of ReduceScatter, instead of lo…
hawkinsp Dec 1, 2023
564bedf
Fix layering violation in ir_emitter_unnested.
jreiffers Dec 1, 2023
8423982
[XLA] [NFC] Generifying hlo-opt to allow easier extension for more pl…
cheshire Dec 1, 2023
51dd5af
Re-enable layering_check for target.
tensorflower-gardener Dec 1, 2023
0367df2
[XLA] Add pass to move cheap fusible computations into while loops to…
blakehechtman Dec 1, 2023
74db590
[XLA] Remove accidental log
blakehechtman Dec 1, 2023
8e41c91
Refactor fused quantization patterns into quantize pass.
chococigar Dec 1, 2023
1ccabf2
Relax coalescing and input fusion heuristics.
jreiffers Dec 1, 2023
52b8b8a
[xla:gpu] Add support for fusing dynamic-update-slice into CUTLASS gemms
ezhulenev Dec 1, 2023
57375f3
[HloValueSemanticsAnalysis] Fix EinsumDepthAnalysis::HandleWhile for …
jinliangwei Dec 1, 2023
7c8072f
adding a unit test for Conv + BiasAdd + Add + <activation> fusion
othakkar Dec 1, 2023
d573615
[stream_executor] Add GetSlice to DeviceMemory*
tyb0807 Dec 1, 2023
19c7beb
Do MLIR verification after each pass.
changm Dec 1, 2023
230bc28
Integrate LLVM at llvm/llvm-project@668865789620
gribozavr Dec 1, 2023
f402286
Set XlaCallModuleOp Attribute jax.uses_shape_polymorphism=true
tensorflower-gardener Dec 1, 2023
f1dcd21
Factor out `EnableDebugging` for `pywrap_` files in quantization.
dansuh17 Dec 1, 2023
aa28566
Update TFRT dependency to use revision
tensorflower-gardener Dec 1, 2023
0d32158
[xla:gpu] Add Async HLO Ops documentation
ezhulenev Dec 1, 2023
50b71dc
[XLA:SPMD] Use CollectiveOpGroupMode::kFlattenedID in GetDefaultColle…
ZixuanJiang Dec 1, 2023
85a5052
[XLA:GPU] Fix crash in triton tiling of pad ops.
jvstokes Dec 1, 2023
7ba21c6
[HloValueSemanticsAnalysis] Add a number of missing handlers.
jinliangwei Dec 1, 2023
50c5cfb
[TSL] Trim dependencies of the CUDA stubs.
hawkinsp Dec 1, 2023
7d6546a
[tflite] Fix tflite selective_build_script.
grantjensen Dec 1, 2023
c1be00f
Remove tensorrt dependency when installing tensorflow[and-cuda]
angerson Dec 1, 2023
1a771bc
[xla:gpu] Add bf16 CUTLASS gemm kernel
ezhulenev Dec 1, 2023
5252fa4
#tf-data Modify the in-memory symbolic checkpointing for Shuffle Op t…
tensorflower-gardener Dec 1, 2023
a93819e
[HloValueSemanticsAnalysis] Add handlers for async host send recv.
jinliangwei Dec 1, 2023
3987ee3
Switch build configs in Linux Arm64 continuous builds from native to …
nitins17 Dec 1, 2023
79c1c93
Replaced std::vector with absl::InlinedVector.
tensorflower-gardener Dec 1, 2023
69fa8c4
Fix missing header inclusion. Found by -Wundefined-func-template.
tkoeppe Dec 1, 2023
aef6b65
No op change.
tensorflower-gardener Dec 1, 2023
f01a271
[HloValueSemanticsAnalysis] Fix activation gradient classification fo…
jinliangwei Dec 1, 2023
2326e80
[XLA] Share the same underlying DFS stack across instructions.
tensorflower-gardener Dec 2, 2023
7052d20
Refactor spmd_partitioner.cc. Merge PatternMatchMergeSharding and Pat…
ZixuanJiang Dec 2, 2023
75d45d3
Lower shape.cstr_broadcastable op in ShapeLegalizeToHLO
tensorflower-gardener Dec 2, 2023
09a463f
Deduplicate stablehlo/experimental passes
GleasonK Dec 2, 2023
598681e
[XLA] Remove hard-coded constants from chi-square test
majnemer Dec 2, 2023
c28fa6e
Add keys() to WeakrefLRUCache. This should probably only be used for …
pschuh Dec 2, 2023
3880d85
[xla:gpu] Add support for custom fusions/kernels to XLA runtime
ezhulenev Dec 2, 2023
0922d15
[xla:gpu] Add kernel launch context to arguments packing to be able t…
ezhulenev Dec 2, 2023
2c537a3
[stream_executor] Replace usage of GetSubBuffer with GetSlice
tyb0807 Dec 2, 2023
47c1593
compat: Update forward compatibility horizon to 2023-12-02
tensorflower-gardener Dec 2, 2023
5d94df8
Update GraphDef version to 1698.
tensorflower-gardener Dec 2, 2023
1fbe954
[CPUExecutable] Add helper to get the name of the main entry point fu…
tensorflower-gardener Dec 2, 2023
7c91223
[XLA] Use operator[] instead of at() for InlinedVector and vector.
tensorflower-gardener Dec 3, 2023
9646aa0
compat: Update forward compatibility horizon to 2023-12-03
tensorflower-gardener Dec 3, 2023
c93b77f
Update GraphDef version to 1699.
tensorflower-gardener Dec 3, 2023
be3cb54
Remove `tpu` tag after quantizing a TPU model.
inho9606 Dec 4, 2023
b043b4e
Adds tests for TransposeOp in `xla_builder_test` and `shape_inference…
yishuangP Dec 4, 2023
567dad7
Add unbounded dynamism for SliceOp.
yishuangP Dec 4, 2023
6261f16
[xla:gpu] Add se::DeviceDescription to custom fusion and fusion matcher
ezhulenev Dec 4, 2023
878a6a0
Merge pull request #62529 from Intel-tensorflow:othakkar/conv_biasadd…
tensorflower-gardener Dec 4, 2023
b100222
PR #7358: [XLA:GPU] Add command buffer memory Free command
shawnwang18 Dec 4, 2023
ac947e9
[XLA:TPU] Fix missing element_size_in_bits coping in Layout Assignmen…
Dec 4, 2023
ae5b2c4
compat: Update forward compatibility horizon to 2023-12-04
tensorflower-gardener Dec 4, 2023
1e44b26
Update GraphDef version to 1700.
tensorflower-gardener Dec 4, 2023
1ed9db0
[XLA] Implement `MHLO` <--> `StableHLO` roundtrip for `TopK` via a cu…
dimitar-asenov Dec 4, 2023
2fc145e
Allow memcpy fusions with multiple copies.
jreiffers Dec 4, 2023
7f62bc6
Fix constant cast in GPU runtime (MemsetImpl)
sergeykozub Dec 4, 2023
5c41c45
[TileAnalysis] Use flat_hash_map to store indexing maps for operands.
pifon2a Dec 4, 2023
f85b281
Priority fusion: fuse bitcasts first.
jreiffers Dec 4, 2023
12a1459
Remove `jit_is_disabled` API since it's not used anywhere and is subs…
yashk2810 Dec 4, 2023
7d4e8ab
PR #7387: [ROCM] build brake fix 231129
pemeliya Dec 4, 2023
2d3b643
[XLA] Fix `Executable::hlo_proto()` to allow a `nullptr` HLO Proto.
dimitar-asenov Dec 4, 2023
4ed1cee
[XLA:GPU] Emit fusions with a single DUS instruction in-place.
olegshyshkov Dec 4, 2023
6f0d8ac
Move xla/python/exceptions.h and xla/python/status_casters.h to pjrt/.
hawkinsp Dec 4, 2023
8d46424
[TileAnalysis] Add indexing computation based on input dimensions.
pifon2a Dec 4, 2023
2cadb32
Changed Version of Bazel to version 6.4.0
tensorflower-gardener Dec 4, 2023
a93f9e9
Deduplicate some code by using the CreateAllStrategiesGroup to genera…
tensorflower-gardener Dec 4, 2023
4ef66d1
[xla:gpu] Add support for CustomFusion matched pattern replacements t…
ezhulenev Dec 4, 2023
659c9ad
Add PjRtTensorBuffer support to GPUUtil::CopyGPUTensorToCPU
SeeForTwo Dec 4, 2023
97afb56
Reorganize code and have separate `AlignedAlloc`, `AlignedFree`, and …
lrdxgm Dec 4, 2023
5c04562
Enable all tests on windows
ddunl Dec 4, 2023
1477082
Use `str | None` rather than `Optional[str]`
ddunl Dec 4, 2023
1e6ad9a
Add logging for DTensorDevice::Execute inputs.
rainwoodman Dec 4, 2023
00b81b7
#tf-data Annotate IO read statistics to xprof to aid user debugging.
wilsingosti Dec 4, 2023
62cc349
[XLA] Make PropagateLivenessThroughTuple O(N).
tensorflower-gardener Dec 4, 2023
ced6245
Create new translate package for graphdef import/export
changm Dec 4, 2023
b650fbd
Various minor cleanups in the test file
tensorflower-gardener Dec 4, 2023
11b1832
PR #7511: [ROCM] build brake fix 23/12/04
pemeliya Dec 4, 2023
fffb030
#tf-data remove the cap and directly use double-typed gauge cell inst…
jimlinntu Dec 4, 2023
d699fc4
[XLA:CPU] Make an experimental dependency on the MLIR GPU dialects op…
hawkinsp Dec 4, 2023
a0f6253
[xla:gpu] Add support for BF16 CUTLASS gemm fusions
ezhulenev Dec 4, 2023
f837e0d
Get rid of a couple unused fields in AutoShardingOption.
tensorflower-gardener Dec 4, 2023
f65a35d
#tf-data Annotate IO read statistics to xprof to aid user debugging.
tensorflower-gardener Dec 4, 2023
f2a4683
[stream_executor] Remove GetSubBuffer
tyb0807 Dec 4, 2023
57c33a2
[xla:ffi] Add XLA_FFI_Error_GetMessage API
ezhulenev Dec 4, 2023
cc8ae1a
Add dialect verification to dialect to verify_input_dialect_to_execut…
rocketas Dec 4, 2023
2e289d3
Refactor the splitting functionality out of XlaSplitNDBaseOp into a s…
deqiangc Dec 4, 2023
6acc044
[stream_executor] Use GetSlice to create sub-buffer
tyb0807 Dec 4, 2023
b92cbae
[stream_executor] Roll back KernelLaunchContext for args packing
ezhulenev Dec 4, 2023
50c4460
[xla:gpu] Add CUTLASS gemm benchmarks
ezhulenev Dec 4, 2023
363c14b
Some more cleanup:
tensorflower-gardener Dec 4, 2023
c909884
Refactor the concatenate functionality out of XlaConcatNDBaseOp into …
deqiangc Dec 5, 2023
dbf1465
When printing a sharding group, also print the index of the correspon…
tensorflower-gardener Dec 5, 2023
1c2e66a
Integrate StableHLO at openxla/stablehlo@57e5a4a5
ghpvnist Dec 5, 2023
e97f5e4
Change seed for a test
majnemer Dec 5, 2023
227a72e
Import ragged_tensor.py in the ragged __init__.py file.
fionalang Dec 5, 2023
7e85121
Add BufferDonor support to cpu_compiler.cc.
pschuh Dec 5, 2023
1308427
Let multi process runner re-raise SkipTest from sub-process.
w-xinyi Dec 5, 2023
83abfdb
Merge pull request #58903 from Intel-tensorflow:mabuzain/hard-swish-f…
tensorflower-gardener Dec 5, 2023
05c3107
Clears the contents of 'solve_info' before computing the solution's f…
tensorflower-gardener Dec 5, 2023
1366242
[XLA] Improve the compile time and memory usage of while_loop_fusible…
blakehechtman Dec 5, 2023
2d016be
[Distributed Eager] Register component functions with a private funct…
mrry Dec 5, 2023
db57943
Migrate experimental macOS x86 nightly builds to the new CI folder
nitins17 Dec 5, 2023
c470219
Merge commit 'db579439eef970657f5ddbf05dc9b798cb748c51' into sync-231206
jayfurmanek Dec 12, 2023
4d21b57
Initial commit to resolve merge conflicts
jayfurmanek Dec 6, 2023
d24bced
Rename `EagerOperation::FunctionDef()` to `EagerOperation::GetFunctio…
mrry Dec 6, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
46 changes: 19 additions & 27 deletions .bazelrc
Original file line number Diff line number Diff line change
Expand Up @@ -561,34 +561,9 @@ build:rbe_linux_rocm_base --platforms="@ubuntu20.04-gcc9_manylinux2014-rocm_conf
build:rbe_linux_rocm_base --action_env=TF_ROCM_CONFIG_REPO="@ubuntu20.04-gcc9_manylinux2014-rocm_config_rocm"
build:rbe_linux_rocm_py3.9 --config=rbe_linux_rocm_base --repo_env=TF_PYTHON_CONFIG_REPO="@ubuntu20.04-gcc9_manylinux2014-rocm_config_python3.9"

build:rbe_linux_cuda_nvcc --config=cuda
build:rbe_linux_cuda_nvcc --action_env=TF_NVCC_CLANG="1"
build:rbe_linux_cuda_nvcc --config=rbe_linux_cuda
build:rbe_linux_cuda_nvcc --repo_env TF_NCCL_USE_STUB=1
build:rbe_linux_cuda_nvcc --@local_xla//xla/python:enable_gpu=true
build:rbe_linux_cuda_nvcc --@local_xla//xla/python:jax_cuda_pip_rpaths=true
build:rbe_linux_cuda_nvcc --define=xla_python_enable_gpu=true
build:rbe_linux_cuda_nvcc --config=tensorrt
build:rbe_linux_cuda_nvcc --repo_env=TF_CUDA_COMPUTE_CAPABILITIES="sm_50,sm_60,sm_70,sm_75,compute_80"
build:rbe_linux_cuda_nvcc --action_env=TF_CUDA_VERSION="12"
build:rbe_linux_cuda_nvcc --action_env=TF_CUDNN_VERSION="8"
build:rbe_linux_cuda_nvcc --action_env=CUDA_TOOLKIT_PATH="/usr/local/cuda-12.2"
build:rbe_linux_cuda_nvcc --action_env=LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64"
build:rbe_linux_cuda_nvcc --crosstool_top="@ubuntu20.04-clang_manylinux2014-cuda12.2-cudnn8.9_config_cuda//crosstool:toolchain"
build:rbe_linux_cuda_nvcc --config=rbe_linux
build:rbe_linux_cuda_nvcc --host_crosstool_top="@ubuntu20.04-clang_manylinux2014-cuda12.2-cudnn8.9_config_cuda//crosstool:toolchain"
build:rbe_linux_cuda_nvcc --extra_toolchains="@ubuntu20.04-clang_manylinux2014-cuda12.2-cudnn8.9_config_cuda//crosstool:toolchain-linux-x86_64"
build:rbe_linux_cuda_nvcc --extra_execution_platforms="@ubuntu20.04-clang_manylinux2014-cuda12.2-cudnn8.9_config_platform//:platform"
build:rbe_linux_cuda_nvcc --host_platform="@ubuntu20.04-clang_manylinux2014-cuda12.2-cudnn8.9_config_platform//:platform"
build:rbe_linux_cuda_nvcc --platforms="@ubuntu20.04-clang_manylinux2014-cuda12.2-cudnn8.9_config_platform//:platform"
build:rbe_linux_cuda_nvcc --repo_env=TF_PYTHON_CONFIG_REPO="@ubuntu20.04-clang_manylinux2014-cuda12.2-cudnn8.9_config_python3.9"
build:rbe_linux_cuda_nvcc --python_path="/usr/bin/python3"
# These you may need to change for your own GCP project.
common:rbe_linux_cuda_nvcc --remote_instance_name=projects/tensorflow-testing/instances/default_instance
build:rbe_linux_cuda_nvcc --repo_env=REMOTE_GPU_TESTING=1
build:rbe_linux_cuda_nvcc --repo_env=TF_CUDA_CONFIG_REPO="@ubuntu20.04-gcc9_manylinux2014-cuda12.2-cudnn8.9_config_cuda"
build:rbe_linux_cuda_nvcc --repo_env=TF_TENSORRT_CONFIG_REPO="@ubuntu20.04-gcc9_manylinux2014-cuda12.2-cudnn8.9_config_tensorrt"
build:rbe_linux_cuda_nvcc --repo_env=TF_NCCL_CONFIG_REPO="@ubuntu20.04-gcc9_manylinux2014-cuda12.2-cudnn8.9_config_nccl"
test:rbe_linux_cuda_nvcc --test_env=LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64"
build:rbe_linux_cuda_nvcc --action_env=TF_NVCC_CLANG="1"

# TODO(kanglan): Remove rbe_win and rbe_win_py3* after b/289091160 is fixed
build:rbe_win --config=rbe_base
Expand Down Expand Up @@ -731,6 +706,15 @@ test:release_cpu_macos --config=release_base
build:release_macos_base --action_env DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer
build:release_macos_base --define=no_nccl_support=true --output_filter=^$

# Build configs for macOS x86
build:release_macos_x86 --config=release_macos_base
# Build with the AVX instruction set when on macOS x86
build:release_macos_x86 --config=avx_linux
build:release_macos_x86 --cpu=darwin
# Target Catalina as the minimum compatible OS version
build:release_macos_x86 --macos_minimum_os=10.15
build:release_macos_x86 --action_env MACOSX_DEPLOYMENT_TARGET=10.15

# Build configs for macOS Arm64
build:release_macos_arm64 --config=release_macos_base
build:release_macos_arm64 --cpu=darwin_arm64
Expand All @@ -745,6 +729,9 @@ test:release_macos_base --test_timeout=300,450,1200,3600 --test_output=errors
test:release_macos_base --build_tests_only --keep_going
test:release_macos_base --flaky_test_attempts=3

# Test configs for macOS x86
test:release_macos_x86 --config=release_macos_base

# Test configs for macOS Arm64
test:release_macos_arm64 --config=release_macos_base

Expand Down Expand Up @@ -806,6 +793,11 @@ test:macos_arm64_wheel_test_filters --test_tag_filters=-no_oss,-oss_excluded,-os
test:macos_arm64_wheel_test_filters --build_tag_filters=-no_oss,-oss_excluded,-oss_serial,-no_oss_py39,-no_oss_py310,-nomac,-no_mac,-mac_excluded,-v1only,-gpu,-tpu,-benchmark-test,-no_mac_arm64,-no_aarch64
test:macos_arm64_wheel_test_filters --test_lang_filters=py --test_size_filters=small,medium
test:macos_arm64_wheel_test --config=macos_arm64_wheel_test_filters -- //tensorflow/... -//tensorflow/python/integration_testing/... -//tensorflow/compiler/tf2tensorrt/... -//tensorflow/core/tpu/... -//tensorflow/lite/... -//tensorflow/tools/toolchains/... -//tensorflow/go/... -//tensorflow/java/... -//tensorflow/compiler/aot/...
# MACOS X86 WHEEL
test:macos_x86_wheel_test_filters --test_tag_filters=-no_oss,-oss_excluded,-oss_serial,-no_oss_py38,-no_oss_py39,-no_oss_py310,-nomac,-no_mac,-mac_excluded,-v1only,-gpu,-tpu,-benchmark-test
test:macos_x86_wheel_test_filters --build_tag_filters=-no_oss,-oss_excluded,-oss_serial,-no_oss_py38,-no_oss_py39,-no_oss_py310,-nomac,-no_mac,-mac_excluded,-v1only,-gpu,-tpu,-benchmark-test
test:macos_x86_wheel_test_filters --test_lang_filters=py --test_size_filters=small,medium
test:macos_x86_wheel_test --config=macos_x86_wheel_test_filters -- //tensorflow/... -//tensorflow/python/integration_testing/... -//tensorflow/compiler/tf2tensorrt/... -//tensorflow/core/tpu/... -//tensorflow/lite/... -//tensorflow/tools/toolchains/... -//tensorflow/go/... -//tensorflow/java/... -//tensorflow/compiler/aot/...

# PYCPP TESTS run a suite of Python and C++ tests to verify general correctness over
# the whole TF code base. These are usually run continuously or upon presubmit.
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/osv-scanner-scheduled.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ permissions:

jobs:
scan-scheduled:
if: github.repository == 'tensorflow/tensorflow'
uses: "google/osv-scanner/.github/workflows/osv-scanner-reusable.yml@main"
with:
scan-args: |-
Expand All @@ -36,4 +37,4 @@ jobs:
--lockfile=requirements.txt:./requirements_lock_3_12.txt
--lockfile=requirements.txt:./ci/official/containers/linux_arm64/devel.requirements.txt
--lockfile=requirements.txt:./ci/official/containers/linux_arm64/jax.requirements.txt
--lockfile=requirements.txt:./ci/official/containers/linux_arm64/devel.usertools/test.requirements.txt
--lockfile=requirements.txt:./ci/official/containers/linux_arm64/devel.usertools/test.requirements.txt
4 changes: 2 additions & 2 deletions .github/workflows/stale-issues.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
pull-requests: write
steps:
- name: Awaiting response issues
uses: actions/stale@v7
uses: actions/stale@6f05e4244c9a0b2ed3401882b05d701dd0a7289b # v7.0.0
with:
#Comma separated list of labels that can be assigned to issues to exclude them from being marked as stale
exempt-issue-labels: 'override-stale'
Expand Down Expand Up @@ -59,7 +59,7 @@ jobs:
close-pr-message: "This PR was closed because it has been inactive for 14 days since being marked as stale. Please reopen if you'd like to work on this further."
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Contribution issues
uses: actions/stale@v7
uses: actions/stale@6f05e4244c9a0b2ed3401882b05d701dd0a7289b # v7.0.0
with:
#Comma separated list of labels that can be assigned to issues to exclude them from being marked as stale
exempt-issue-labels: 'override-stale'
Expand Down
6 changes: 6 additions & 0 deletions RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@
* <DOCUMENT BREAKING CHANGES HERE>
* <THIS SECTION SHOULD CONTAIN API, ABI AND BEHAVIORAL BREAKING CHANGES>

* `tf.summary.trace_on` now takes a `profiler_outdir` argument. This must be set
if `profiler` arg is set to `True`.
* `tf.summary.trace_export`'s `profiler_outdir` arg is now a no-op. Enabling
the profiler now requires setting `profiler_outdir` in `trace_on`.


### Known Caveats

* <CAVEATS REGARDING THE RELEASE (BUT NOT BREAKING CHANGES).>
Expand Down
7 changes: 6 additions & 1 deletion ci/official/envs/ci_default
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,9 @@ TFCI_WHL_AUDIT_PLAT=
TFCI_WHL_BAZEL_TEST_ENABLE=1
TFCI_WHL_SIZE_LIMIT=
TFCI_WHL_SIZE_LIMIT_ENABLE=1
TFCI_PYENV_INSTALL_LOCAL_ENABLE=
TFCI_MACOS_UPGRADE_PYENV_ENABLE=
TFCI_MACOS_INSTALL_BAZELISK_ENABLE=
TFCI_MACOS_INSTALL_BAZELISK_URL=
TFCI_MACOS_PYENV_INSTALL_ENABLE=
TFCI_MACOS_BAZEL_TEST_DIR_ENABLE=
TFCI_MACOS_BAZEL_TEST_DIR_PATH=
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# This envrionment is experimental and should not yet be used for production jobs
TFCI_BAZEL_COMMON_ARGS="--config rbe_cross_compile_linux_arm64 --repo_env=TF_PYTHON_VERSION=$TFCI_PYTHON_VERSION"
TFCI_BAZEL_TARGET_SELECTING_CONFIG_PREFIX=cross_compile_linux_arm64
TFCI_DOCKER_IMAGE=gcr.io/tensorflow-sigs/build-arm64:tf-latest-multi-python
TFCI_DOCKER_REBUILD_ARGS="--target=tf ci/official/containers/linux_arm64"
TFCI_PYTHON_VERSION=3.9
1 change: 1 addition & 0 deletions ci/official/envs/continuous_macos_arm64_py310
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ TFCI_BAZEL_COMMON_ARGS="--config release_macos_arm64 --config tf_public_macos_ca
TFCI_BAZEL_TARGET_SELECTING_CONFIG_PREFIX=macos_arm64
TFCI_DOCKER_ENABLE=0
TFCI_PYTHON_VERSION=3.10
TFCI_MACOS_BAZEL_TEST_DIR_PATH="/Volumes/BuildData/bazel_output"
1 change: 1 addition & 0 deletions ci/official/envs/continuous_macos_arm64_py311
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ TFCI_BAZEL_COMMON_ARGS="--config release_macos_arm64 --config tf_public_macos_ca
TFCI_BAZEL_TARGET_SELECTING_CONFIG_PREFIX=macos_arm64
TFCI_DOCKER_ENABLE=0
TFCI_PYTHON_VERSION=3.11
TFCI_MACOS_BAZEL_TEST_DIR_PATH="/Volumes/BuildData/bazel_output"
1 change: 1 addition & 0 deletions ci/official/envs/continuous_macos_arm64_py39
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ TFCI_BAZEL_COMMON_ARGS="--config release_macos_arm64 --config tf_public_macos_ca
TFCI_BAZEL_TARGET_SELECTING_CONFIG_PREFIX=macos_arm64
TFCI_DOCKER_ENABLE=0
TFCI_PYTHON_VERSION=3.9
TFCI_MACOS_BAZEL_TEST_DIR_PATH="/Volumes/BuildData/bazel_output"
8 changes: 4 additions & 4 deletions ci/official/envs/nightly_libtensorflow_macos_arm64
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
source ci/official/envs/ci_nightly_uploads
TFCI_BAZEL_COMMON_ARGS="--config release_macos_arm64 --config tf_public_cache_push --config resultstore --repo_env=TF_PYTHON_VERSION=$TFCI_PYTHON_VERSION"
# Disable arm64 uploads while being worked on
source ci/official/envs/disable_all_uploads
TFCI_BAZEL_COMMON_ARGS="--config release_macos_arm64 --config tf_public_cache_push --repo_env=TF_PYTHON_VERSION=$TFCI_PYTHON_VERSION"
TFCI_DOCKER_ENABLE=0
TFCI_LIB_SUFFIX="-cpu-macos-arm64"
TFCI_LIB_SUFFIX="-cpu-darwin-arm64"
TFCI_NIGHTLY_UPDATE_VERSION_ENABLE=1
TFCI_PYTHON_VERSION=3.10
TFCI_UPLOAD_WHL_GCS_URI=1
7 changes: 7 additions & 0 deletions ci/official/envs/nightly_libtensorflow_macos_x86
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Disable macOS x86 uploads while being worked on
source ci/official/envs/disable_all_uploads
TFCI_BAZEL_COMMON_ARGS="--config release_macos_x86 --config tf_public_cache_push --repo_env=TF_PYTHON_VERSION=$TFCI_PYTHON_VERSION"
TFCI_DOCKER_ENABLE=0
TFCI_LIB_SUFFIX="-cpu-darwin-x86_64"
TFCI_NIGHTLY_UPDATE_VERSION_ENABLE=1
TFCI_PYTHON_VERSION=3.10
5 changes: 2 additions & 3 deletions ci/official/envs/nightly_macos_arm64_py310
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@ TFCI_BUILD_PIP_PACKAGE_ARGS="--cpu --nightly_flag"
TFCI_DOCKER_ENABLE=0
TFCI_NIGHTLY_UPDATE_VERSION_ENABLE=1
TFCI_PYTHON_VERSION=3.10
TFCI_UPLOAD_WHL_GCS_ENABLE=1
TFCI_UPLOAD_WHL_GCS_URI="gs://tensorflow-nightly/macos-arm64/$(date -I)"
TFCI_WHL_AUDIT_ENABLE=
TFCI_WHL_SIZE_LIMIT=240M
TFCI_PYENV_INSTALL_LOCAL_ENABLE=$TFCI_PYTHON_VERSION
TFCI_MACOS_PYENV_INSTALL_ENABLE=1
TFCI_MACOS_BAZEL_TEST_DIR_PATH="/Volumes/BuildData/bazel_output"
4 changes: 1 addition & 3 deletions ci/official/envs/nightly_macos_arm64_py311
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@ TFCI_BUILD_PIP_PACKAGE_ARGS="--cpu --nightly_flag"
TFCI_DOCKER_ENABLE=0
TFCI_NIGHTLY_UPDATE_VERSION_ENABLE=1
TFCI_PYTHON_VERSION=3.11
TFCI_UPLOAD_WHL_GCS_ENABLE=1
TFCI_UPLOAD_WHL_GCS_URI="gs://tensorflow-nightly/macos-arm64/$(date -I)"
TFCI_WHL_AUDIT_ENABLE=
TFCI_WHL_SIZE_LIMIT=240M
TFCI_PYENV_INSTALL_LOCAL_ENABLE=$TFCI_PYTHON_VERSION
TFCI_MACOS_BAZEL_TEST_DIR_PATH="/Volumes/BuildData/bazel_output"
5 changes: 2 additions & 3 deletions ci/official/envs/nightly_macos_arm64_py312
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@ TFCI_BUILD_PIP_PACKAGE_ARGS="--cpu --nightly_flag"
TFCI_DOCKER_ENABLE=0
TFCI_NIGHTLY_UPDATE_VERSION_ENABLE=1
TFCI_PYTHON_VERSION=3.12
TFCI_UPLOAD_WHL_GCS_ENABLE=
TFCI_UPLOAD_WHL_GCS_URI="gs://tensorflow-nightly/macos-arm64/$(date -I)"
TFCI_WHL_AUDIT_ENABLE=
TFCI_WHL_SIZE_LIMIT=240M
TFCI_PYENV_INSTALL_LOCAL_ENABLE=$TFCI_PYTHON_VERSION
TFCI_MACOS_PYENV_INSTALL_ENABLE=1
TFCI_MACOS_BAZEL_TEST_DIR_PATH="/Volumes/BuildData/bazel_output"
5 changes: 2 additions & 3 deletions ci/official/envs/nightly_macos_arm64_py39
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@ TFCI_BUILD_PIP_PACKAGE_ARGS="--cpu --nightly_flag"
TFCI_DOCKER_ENABLE=0
TFCI_NIGHTLY_UPDATE_VERSION_ENABLE=1
TFCI_PYTHON_VERSION=3.9
TFCI_UPLOAD_WHL_GCS_ENABLE=1
TFCI_UPLOAD_WHL_GCS_URI="gs://tensorflow-nightly/macos/arm64/$(date -I)"
TFCI_WHL_AUDIT_ENABLE=
TFCI_WHL_SIZE_LIMIT=240M
TFCI_PYENV_INSTALL_LOCAL_ENABLE=$TFCI_PYTHON_VERSION
TFCI_MACOS_PYENV_INSTALL_ENABLE=1
TFCI_MACOS_BAZEL_TEST_DIR_PATH="/Volumes/BuildData/bazel_output"
16 changes: 16 additions & 0 deletions ci/official/envs/nightly_macos_x86_py310
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Disable macOS x86 uploads while being worked on
source ci/official/envs/disable_all_uploads
TFCI_BAZEL_COMMON_ARGS="--config release_macos_x86 --config tf_public_macos_cache_push --repo_env=TF_PYTHON_VERSION=$TFCI_PYTHON_VERSION"
TFCI_BAZEL_TARGET_SELECTING_CONFIG_PREFIX=macos_x86
TFCI_BUILD_PIP_PACKAGE_ARGS="--cpu --nightly_flag"
TFCI_DOCKER_ENABLE=0
TFCI_NIGHTLY_UPDATE_VERSION_ENABLE=1
TFCI_PYTHON_VERSION=3.10
TFCI_WHL_AUDIT_ENABLE=
TFCI_WHL_SIZE_LIMIT=255M
TFCI_MACOS_INSTALL_BAZELISK_ENABLE=1
TFCI_MACOS_INSTALL_BAZELISK_URL="https://github.com/bazelbuild/bazelisk/releases/download/v1.11.0/bazelisk-darwin-amd64"
TFCI_MACOS_UPGRADE_PYENV_ENABLE=1
TFCI_MACOS_PYENV_INSTALL_ENABLE=1
TFCI_MACOS_BAZEL_TEST_DIR_ENABLE=1
TFCI_MACOS_BAZEL_TEST_DIR_PATH="/Volumes/BuildData/bazel_output"
16 changes: 16 additions & 0 deletions ci/official/envs/nightly_macos_x86_py311
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Disable macOS x86 uploads while being worked on
source ci/official/envs/disable_all_uploads
TFCI_BAZEL_COMMON_ARGS="--config release_macos_x86 --config tf_public_macos_cache_push --repo_env=TF_PYTHON_VERSION=$TFCI_PYTHON_VERSION"
TFCI_BAZEL_TARGET_SELECTING_CONFIG_PREFIX=macos_x86
TFCI_BUILD_PIP_PACKAGE_ARGS="--cpu --nightly_flag"
TFCI_DOCKER_ENABLE=0
TFCI_NIGHTLY_UPDATE_VERSION_ENABLE=1
TFCI_PYTHON_VERSION=3.11
TFCI_WHL_AUDIT_ENABLE=
TFCI_WHL_SIZE_LIMIT=255M
TFCI_MACOS_INSTALL_BAZELISK_ENABLE=1
TFCI_MACOS_INSTALL_BAZELISK_URL="https://github.com/bazelbuild/bazelisk/releases/download/v1.11.0/bazelisk-darwin-amd64"
TFCI_MACOS_UPGRADE_PYENV_ENABLE=1
TFCI_MACOS_PYENV_INSTALL_ENABLE=1
TFCI_MACOS_BAZEL_TEST_DIR_ENABLE=1
TFCI_MACOS_BAZEL_TEST_DIR_PATH="/Volumes/BuildData/bazel_output"
16 changes: 16 additions & 0 deletions ci/official/envs/nightly_macos_x86_py312
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Disable macOS x86 uploads while being worked on
source ci/official/envs/disable_all_uploads
TFCI_BAZEL_COMMON_ARGS="--config release_macos_x86 --config tf_public_macos_cache_push --repo_env=TF_PYTHON_VERSION=$TFCI_PYTHON_VERSION"
TFCI_BAZEL_TARGET_SELECTING_CONFIG_PREFIX=macos_x86
TFCI_BUILD_PIP_PACKAGE_ARGS="--cpu --nightly_flag"
TFCI_DOCKER_ENABLE=0
TFCI_NIGHTLY_UPDATE_VERSION_ENABLE=1
TFCI_PYTHON_VERSION=3.12
TFCI_WHL_AUDIT_ENABLE=
TFCI_WHL_SIZE_LIMIT=255M
TFCI_MACOS_INSTALL_BAZELISK_ENABLE=1
TFCI_MACOS_INSTALL_BAZELISK_URL="https://github.com/bazelbuild/bazelisk/releases/download/v1.11.0/bazelisk-darwin-amd64"
TFCI_MACOS_UPGRADE_PYENV_ENABLE=1
TFCI_MACOS_PYENV_INSTALL_ENABLE=1
TFCI_MACOS_BAZEL_TEST_DIR_ENABLE=1
TFCI_MACOS_BAZEL_TEST_DIR_PATH="/Volumes/BuildData/bazel_output"
14 changes: 14 additions & 0 deletions ci/official/envs/nightly_macos_x86_py39
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Disable macOS x86 uploads while being worked on
source ci/official/envs/disable_all_uploads
TFCI_BAZEL_COMMON_ARGS="--config release_macos_x86 --config tf_public_macos_cache_push --repo_env=TF_PYTHON_VERSION=$TFCI_PYTHON_VERSION"
TFCI_BAZEL_TARGET_SELECTING_CONFIG_PREFIX=macos_x86
TFCI_BUILD_PIP_PACKAGE_ARGS="--cpu --nightly_flag"
TFCI_DOCKER_ENABLE=0
TFCI_NIGHTLY_UPDATE_VERSION_ENABLE=1
TFCI_PYTHON_VERSION=3.9
TFCI_WHL_AUDIT_ENABLE=
TFCI_WHL_SIZE_LIMIT=255M
TFCI_MACOS_INSTALL_BAZELISK_ENABLE=1
TFCI_MACOS_INSTALL_BAZELISK_URL="https://github.com/bazelbuild/bazelisk/releases/download/v1.11.0/bazelisk-darwin-amd64"
TFCI_MACOS_BAZEL_TEST_DIR_ENABLE=1
TFCI_MACOS_BAZEL_TEST_DIR_PATH="/Volumes/BuildData/bazel_output"
70 changes: 50 additions & 20 deletions ci/official/utilities/setup_macos.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,31 +34,61 @@ else
exit 1
fi

if [[ -n "${KOKORO_JOB_NAME}" ]]; then
# Mac builds need ~150 GB of disk space to be able to run all the tests. By
# default, Kokoro runs the Bazel commands in a partition that does not have
# enough free space so we need to set TEST_TMPDIR explicitly.
mkdir -p /Volumes/BuildData/bazel_output
export TEST_TMPDIR=/Volumes/BuildData/bazel_output
# "TFCI_MACOS_BAZEL_TEST_DIR_PATH" specifies the directory that Bazel should use
# when running tests. Each test will be executed in a separate subdirectory
# inside this directory. TF Mac builds need ~150 GB of disk space to be able to
# run all the tests. Since TFCI Mac VMs execute Bazel test commands in a
# partition with insufficient storage, we specify the
# 'TFCI_MACOS_BAZEL_TEST_DIR_PATH' environment variable to point to a partition
# with ample storage. When this variable is empty (i.e by default), Bazel will
# use the output base directory to run tests.
if [[ "${TFCI_MACOS_BAZEL_TEST_DIR_ENABLE}" == 1 ]]; then
mkdir -p "${TFCI_MACOS_BAZEL_TEST_DIR_PATH}"
export TEST_TMPDIR="${TFCI_MACOS_BAZEL_TEST_DIR_PATH}"
fi

# "TFCI_MACOS_INSTALL_BAZELISK_ENABLE" is used to decide if we need to install
# Bazelisk manually. We enable this for macOS x86 builds as those VMs do not
# have Bazelisk pre-installed. "TFCI_MACOS_INSTALL_BAZELISK_URL" contains the
# link to the Bazelisk binary which needs to be downloaded.
if [[ "${TFCI_MACOS_INSTALL_BAZELISK_ENABLE}" == 1 ]]; then
sudo wget --no-verbose -O "/usr/local/bin/bazel" "${TFCI_MACOS_INSTALL_BAZELISK_URL}"
chmod +x "/usr/local/bin/bazel"
fi

# Before uploading the nightly and release wheels, we install them in a
# virtual environment and run some smoke tests on it. The Kokoro Mac VMs
# only have Python 3.11 installed so we need to install the other Python
# versions manually.
if [[ -n "${TFCI_BUILD_PIP_PACKAGE_ARGS}" ]] && [[ "${TFCI_PYENV_INSTALL_LOCAL_ENABLE}" != 3.11 ]]; then
pyenv install "${TFCI_PYENV_INSTALL_LOCAL_ENABLE}"
pyenv local "${TFCI_PYENV_INSTALL_LOCAL_ENABLE}"
fi
elif [[ "${TFCI_WHL_BAZEL_TEST_ENABLE}" == 1 ]]; then
echo '==TFCI==: Note: Mac builds need ~150 GB of disk space to be able to'
echo 'run all the tests. Please make sure your system has enough disk space'
echo 'You can control where Bazel stores test artifacts by setting the'
echo '`TEST_TMPDIR` environment variable.'
# "TFCI_MACOS_UPGRADE_PYENV_ENABLE" is used to decide if we need to upgrade the
# Pyenv version. We enable this for macOS x86 builds as the default Pyenv on
# those VMs does not support installing Python 3.10 and above which we need
# for running smoke tests in nightly/release wheel builds.
if [[ "${TFCI_MACOS_UPGRADE_PYENV_ENABLE}" == 1 ]]; then
brew upgrade pyenv
fi

if [[ "${TFCI_PYTHON_VERSION}" == "3.12" ]]; then
# "TFCI_MACOS_PYENV_INSTALL_ENABLE" controls whether to use Pyenv to install
# the Python version set in "TFCI_PYTHON_VERSION" and use it as default.
# We enable this in the nightly and release builds because before uploading the
# wheels, we install them in a virtual environment and run some smoke tests on
# it. TFCI Mac VMs only have one Python version installed so we need to install
# the other versions manually.
if [[ "${TFCI_MACOS_PYENV_INSTALL_ENABLE}" == 1 ]]; then
pyenv install "$TFCI_PYTHON_VERSION"
pyenv local "$TFCI_PYTHON_VERSION"
# Do a sanity check to make sure that we using the correct Python version
python --version
fi

if [[ "$TFCI_PYTHON_VERSION" == "3.12" ]]; then
# dm-tree (Keras v3 dependency) doesn't have pre-built wheels for 3.12 yet.
# Having CMake allows building them.
# Once the wheels are added, this should be removed - b/308399490.
brew install cmake
fi

# Scheduled nightly and release builds upload build artifacts (Pip packages,
# Libtensorflow archives) to GCS buckets. TFCI Mac VMs need to authenticate as
# a service account that has the right permissions to be able to do so.
set +x
if [[ -n "${GOOGLE_APPLICATION_CREDENTIALS:-}" ]]; then
gcloud auth activate-service-account
fi
set -x
Loading